Method and system for measuring user engagement using click/skip in content stream

ABSTRACT

Method, system, and programs for measuring user engagement. In one example, a model generated based on user activities with respect to a plurality pieces of content is obtained. One or more actual occurrences of the user activities with respect to one piece of the plurality pieces of content are identified. One or more future occurrences of the user activities with respect to the piece of content are estimated based on the model. A user engagement score with respect to the piece of content is calculated based on the one or more actual occurrences of the user activities and the one or more future occurrences of the user activities.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent applicationSer. No. 13/836,556, filed Mar. 15, 2013, which is incorporated hereinby reference in its entirety.

BACKGROUND Technical Field

The present teaching relates to methods and systems for providingcontent. Specifically, the present teaching relates to methods andsystems for providing online content.

Discussion of Technical Background

The Internet has made it possible for a user to electronically accessvirtually any content at anytime and from any location. With theexplosion of information, it has become more and more important toprovide users with information that is relevant to the user and not justinformation in general. Further, as users of today's society rely on theInternet as their source of information, entertainment, and/or socialconnections, e.g., news, social interaction, movies, music, etc, it iscritical to provide users with information they find valuable.

Efforts have been made to attempt to allow users to readily accessrelevant and on the point content. For example, topical portals havebeen developed that are more subject matter oriented as compared togeneric content gathering systems such as traditional search engines.Example topical portals include portals on finance, sports, news,weather, shopping, music, art, film, etc. Such topical portals allowusers to access information related to subject matters that theseportals are directed to. Users have to go to different portals to accesscontent of certain subject matter, which is not convenient and not usercentric.

Another line of efforts in attempting to enable users to easily accessrelevant content is via personalization, which aims at understandingeach user's individual likings/interests/preferences so that anindividualized user profile for each user can be set up and can be usedto select content that matches a user's interests. The underlying goalis to meet the minds of users in terms of content consumption. Userprofiles traditionally are constructed based on users' declaredinterests and/or inferred from, e.g., users' demographics. There havealso been systems that identify users' interests based on observationsmade on users' interactions with content. A typical example of such userinteraction with content is click through rate (CTR).

These traditional approaches have various shortcomings. For example,users' interests are profiled without any reference to a baseline sothat the level of interest can be more accurately estimated. Userinterests are detected in isolated application settings so that userprofiling in individual applications cannot capture a broad range of theoverall interests of a user. Such traditional approach to user profilinglead to fragmented representation of user interests without a coherentunderstanding of the users' preferences. Because profiles of the sameuser derived from different application settings are often grounded withrespect to the specifics of the applications, it is also difficult tointegrate them to generate a more coherent profile that better representthe user's interests.

User activities directed to content are traditionally observed and usedto estimate or infer users' interests. CTR is the most commonly usedmeasure to estimate users' interests. However, CTR is no longer adequateto capture users' interests particularly given that different types ofactivities that a user may perform on different types of devices mayalso reflect or implicate user's interests. In addition, user reactionsto content usually represent users' short term interests. Such observedshort term interests, when acquired piece meal, as traditionalapproaches often do, can only lead to reactive, rather than proactive,services to users. Although short term interests are important, they arenot adequate to enable understanding of the more persistent long terminterests of a user, which are crucial in terms of user retention. Mostuser interactions with content represent short term interests of theuser so that relying on such short term interest behavior makes itdifficult to expand the understanding of the increasing range ofinterests of the user. When this is in combination with the fact thatsuch collected data is always the past behavior and collected passively,it creates a personalization bubble, making it difficult, if notimpossible, to discover other interests of a user unless the userinitiates some action to reveal new interests.

Yet another line of effort to allow users to access relevant content isto pooling content that may be interested by users in accordance withtheir interests. Given the explosion of information on the Internet, itis not likely, even if possible, to evaluate all content accessible viathe Internet whenever there is a need to select content relevant to aparticular user. Thus, realistically, it is needed to identify a subsetor a pool of the Internet content based on some criteria so that contentcan be selected from this pool and recommended to users based on theirinterests for consumption.

Conventional approaches to creating such a subset of content areapplication centric. Each application carves out its own subset ofcontent in a manner that is specific to the application. For example,Amazon.com may have a content pool related to products and informationassociated thereof created/updated based on information related to itsown users and/or interests of such users exhibited when they interactwith Amazon.com. Facebook also has its own subset of content, generatedin a manner not only specific to Facebook but also based on userinterests exhibited while they are active on Facebook. As a user may beactive in different applications (e.g., Amazon.com and Facebook) andwith each application, they likely exhibit only part of their overallinterests in connection with the nature of the application. Given that,each application can usually gain understanding, at best, of partialinterests of users, making it difficult to develop a subset of contentthat can be used to serve a broader range of users' interests.

Another line of effort is directed to personalized contentrecommendation, i.e., selecting content from a content pool based on theuser's personalized profiles and recommending such identified content tothe user. Conventional solutions focus on relevance, i.e., the relevancebetween the content and the user. Although relevance is important, thereare other factors that also impact how recommendation content should beselected in order to satisfy a user's interests. Most contentrecommendation systems insert advertisement to content identified for auser for recommendation. Some traditional systems that are used toidentify insertion advertisements match content with advertisement oruser's query (also content) with advertisement, without consideringmatching based on demographics of the user with features of the targetaudience defined by advertisers. Some traditional systems match userprofiles with the specified demographics of the target audience definedby advertisers but without matching the content to be provided to theuser and the advertisement. The reason is that content is oftenclassified into taxonomy based on subject matters covered in the contentyet advertisement taxonomy is often based on desired target audiencegroups. This makes it less effective in terms of selecting the mostrelevant advertisement to be inserted into content to be recommended toa specific user.

There is a need for improvements over the conventional approaches topersonalizing content recommendation.

SUMMARY

The present teaching relates to methods, systems, and programming formeasuring user engagement. Particularly, the present teaching relates tomethods, systems, and programming for measuring user engagement inpersonalized content recommendation.

In one example, a method, implemented on at least one machine each ofwhich has at least one processor, storage, and a communication platformconnected to a network for measuring user engagement, is disclosed. Amodel generated based on user activities with respect to a pluralitypieces of content is obtained. One or more actual occurrences of theuser activities with respect to one piece of the plurality pieces ofcontent are identified. One or more future occurrences of the useractivities with respect to the piece of content are estimated based onthe model. A user engagement score with respect to the piece of contentis calculated based on the one or more actual occurrences of the useractivities and the one or more future occurrences of the useractivities.

In a different example, a system for measuring user engagement isdisclosed. The system includes a model building unit, a user activitydetection module, and a user engagement score calculation unit. Themodel building unit is configured to generate a model based on useractivities with respect to a plurality pieces of content. The useractivity detection module is configured to identify one or more actualoccurrences of the user activities with respect to one piece of theplurality pieces of content. The user engagement score calculation unitis configured to estimate one or more future occurrences of the useractivities with respect to the piece of content based on the model. Theuser engagement score calculation unit is also configured to calculate auser engagement score with respect to the piece of content based on theone or more actual occurrences of the user activities and the one ormore future occurrences of the user activities.

Other concepts relate to software for measuring user engagement. Asoftware product, in accord with this concept, includes at least onemachine-readable non-transitory medium and information carried by themedium. The information carried by the medium may be executable programcode data regarding parameters in association with a request oroperational parameters, such as information related to a user, arequest, or a social group, etc.

In one example, a machine readable and non-transitory medium havinginformation recorded thereon for measuring user engagement, wherein theinformation, when read by the machine, causes the machine to perform aseries of steps. A model is gene rated based on user activities withrespect to a plurality pieces of content. One or more actual occurrencesof the user activities with respect to one piece of the plurality piecesof content are identified. One or more future occurrences of the useractivities with respect to the piece of content are estimated based onthe model. A user engagement score with respect to the piece of contentis calculated based on the one or more actual occurrences of the useractivities and the one or more future occurrences of the useractivities.

BRIEF DESCRIPTION OF THE DRAWINGS

The methods, systems, and/or programming described herein are furtherdescribed in terms of exemplary embodiments. These exemplary embodimentsare described in detail with reference to the drawings. Theseembodiments are non-limiting exemplary embodiments, in which likereference numerals represent similar structures throughout the severalviews of the drawings, and wherein:

FIG. 1 depicts an exemplary system diagram for personalized contentrecommendation, according to an embodiment of the present teaching;

FIG. 2 is a flowchart of an exemplary process for personalized contentrecommendation, according to an embodiment of the present teaching;

FIG. 3 illustrates exemplary types of context information;

FIG. 4 depicts an exemplary diagram of a content pool generation/updateunit, according to an embodiment of the present teaching;

FIG. 5 is a flowchart of an exemplary process of creating a contentpool, according to an embodiment of the present teaching;

FIG. 6 is a flowchart of an exemplary process for updating a contentpool, according to an embodiment of the present teaching;

FIG. 7 depicts an exemplary diagram of a user understanding unit,according to an embodiment of the present teaching;

FIG. 8 is a flowchart of an exemplary process for generating a baselineinterest profile, according to an embodiment of the present teaching;

FIG. 9 is a flowchart of an exemplary process for generating apersonalized user profile, according to an embodiment of the presentteaching;

FIG. 10 depicts an exemplary system diagram for a content ranking unit,according to an embodiment of the present teaching;

FIG. 11 is a flowchart of an exemplary process for the content rankingunit, according to an embodiment of the present teaching;

FIG. 12 is a high level exemplary system diagram of a user engagementassessment system, according to an embodiment of the present teaching;

FIG. 13 depicts exemplary applications of user engagement measures,according to different embodiments of the present teaching;

FIG. 14 is a function block diagram of one example of the userengagement assessment system shown in FIG. 12, according to anembodiment of the present teaching;

FIGS. 15-16 are flowcharts of an exemplary process of the userengagement assessment system shown in FIG. 14, according to differentembodiments of the present teaching;

FIG. 17 is a function block diagram of another example of the userengagement assessment system shown in FIG. 12, according to anembodiment of the present teaching;

FIG. 18 is a flowchart of an exemplary process of the user engagementassessment system shown in FIG. 17, according to an embodiment of thepresent teaching;

FIG. 19 illustrates an exemplary function of measuring user engagementbased on stream depth, according to an embodiment of the presentteaching;

FIG. 20 is a function block diagram of still another example of the userengagement assessment system shown in FIG. 12, according to anembodiment of the present teaching;

FIG. 21 is a flowchart of an exemplary process of the user engagementassessment system shown in FIG. 20, according to an embodiment of thepresent teaching;

FIG. 22 depicts an exemplary content stream presented in different timeperiods, according to an embodiment of the present teaching;

FIGS. 23-25 illustrate exemplary user activities with respect tomultiple pieces of content in a content stream, according to differentembodiments of the present teaching;

FIGS. 26-28 depict exemplary embodiments of a networked environment inwhich user engagement measurement is applied, according to differentembodiments of the present teaching;

FIG. 29 depicts a general mobile device architecture on which thepresent teaching can be implemented; and

FIG. 30 depicts a general computer architecture on which the presentteaching can be implemented.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, well known methods, procedures, components,and/or circuitry have been described at a relatively high-level, withoutdetail, in order to avoid unnecessarily obscuring aspects of the presentteachings.

The present teaching relates to personalizing on-line contentrecommendations to a user. Particularly, the present teaching relates toa system, method, and/or programs for personalized contentrecommendation that addresses the shortcomings associated theconventional content recommendation solutions in personalization,content pooling, and recommending personalized content.

With regard to personalization, the present teaching identifies a user'sinterests with respect to a universal interest space, defined via knownconcept archives such as Wikipedia and/or content taxonomy. Using such auniversal interest space, interests of users, exhibited in differentapplications and via different platforms, can be used to establish ageneral population's profile as a baseline against which individualuser's interests and levels thereof can be determined. For example,users active in a third party application such as Facebook or Twitterand the interests that such users exhibited in these third partyapplications can be all mapped to the universal interest space and thenused to compute a baseline interest profile of the general population.Specifically, each user's interests observed with respect to eachdocument covering certain subject matters or concepts can be mapped to,e.g., Wikipedia or certain content taxonomy. A high dimensional vectorcan be constructed based on the universal interest space in which eachattribute of the vector corresponds to a concept in the universal spaceand the value of the attribute may corresponds to an evaluation of theuser's interest in this particular concept. The general baselineinterest profile can be derived based on all vectors represent thepopulation. Each vector representing an individual can be normalizedagainst the baseline interest profile so that the relative level ofinterests of the user with respect to the concepts in the universalinterest space can be determined. This enables better understanding ofthe level of interests of the user in different subject matters withrespect to a more general population and result in enhancedpersonalization for content recommendation. Rather than characterizingusers' interests merely according to proprietary content taxonomy, as isoften done in the prior art, the present teaching leverages publicconcept archives, such as Wikipedia or online encyclopedia, to define auniversal interest space in order to profile a user's interests in amore coherent manner. Such a high dimensional vector captures the entireinterest space of every user, making person-to-person comparison as topersonal interests more effective. Profiling a user and in this manneralso leads to efficient identification of users who share similarinterests. In addition, content may also be characterized in the sameuniversal interest space, e.g., a high dimensional vector against theconcepts in the universal interest space can also be constructed withvalues in the vector indicating whether the content covers each of theconcepts in the universal interest space. By characterizing users andcontent in the same space in a coherent way, the affinity between a userand a piece of content can be determined via, e.g., a dot product of thevector for the user and the vector for the content.

The present teaching also leverages short term interests to betterunderstand long term interests of users. Short term interests can beobserved via user online activities and used in online contentrecommendation, the more persistent long term interests of a user canhelp to improve content recommendation quality in a more robust mannerand, hence, user retention rate. The present teaching disclosesdiscovery of long term interests as well as short term interests.

To improve personalization, the present teaching also discloses ways toimprove the ability to estimate a user's interest based on a variety ofuser activities. This is especially useful because meaningful useractivities often occur in different settings, on different devices, andin different operation modes. Through such different user activities,user engagement to content can be measured to infer users' interests.Traditionally, clicks and click through rate (CTR) have been used toestimate users' intent and infer users' interests. CTR is simply notadequate in today's world. Users may dwell on a certain portion of thecontent, the dwelling may be for different lengths of time, users mayscroll along the content and may dwell on a specific portion of thecontent for some length of time, users may scroll down at differentspeeds, users may change such speed near certain portions of content,users may skip certain portion of content, etc. All such activities mayhave implications as to users' engagement to content. Such engagementcan be utilized to infer or estimate a user's interests. The presentteaching leverages a variety of user activities that may occur acrossdifferent device types in different settings to achieve betterestimation of users' engagement in order to enhance the ability ofcapturing a user's interests in a more reliable manner.

Another aspect of the present teaching with regard to personalization isits ability to explore unknown interests of a user by generating probingcontent. Traditionally, user profiling is based on either user providedinformation (e.g., declared interests) or passively observed pastinformation such as the content that the user has viewed, reactions tosuch content, etc. Such prior art schemes can lead to a personalizationbubble where only interests that the user revealed can be used forcontent recommendation. Because of that, the only user activities thatcan be observed are directed to such known interests, impeding theability to understand the overall interest of a user. This is especiallyso considering the fact that users often exhibit different interests(mostly partial interests) in different application settings. Thepresent teaching discloses ways to generate probing content withconcepts that is currently not recognized as one of the user's interestsin order to explore the user's unknown interests. Such probing contentis selected and recommended to the user and user activities directed tothe probing content can then be analyzed to estimate whether the userhas other interests. The selection of such probing content may be basedon a user's current known interests by, e.g., extrapolating the user'scurrent interests. For example, for some known interests of the user(e.g., the short term interests at the moment), some probing concepts inthe universal interest space, for which the user has not exhibitedinterests in the past, may be selected according to some criteria (e.g.,within a certain distance from the user's current known interest in ataxonomy tree) and content related to such probing concepts may then beselected and recommended to the user. Another way to identify probingconcept (corresponding to unknown interest of the user) may be throughthe user's cohorts. For instance, a user may share certain interestswith his/her cohorts but some members of the circle may have someinterests that the user has never exhibited before. Such un-sharedinterests with cohorts may be selected as probing unknown interests forthe user and content related to such probing unknown interests may thenbe selected as probing content to be recommended to the user. In thismanner, the present teaching discloses a scheme by which a user'sinterests can be continually probed and understood to improve thequality of personalization. Such managed probing can also be combinedwith random selection of probing content to allow discovery of unknowninterests of the user that are far removed from the user's current knowninterests.

A second aspect of recommending quality personalized content is to builda content pool with quality content that covers subject mattersinteresting to users. Content in the content pool can be rated in termsof the subject and/or the performance of the content itself. Forexample, content can be characterized in terms of concepts it disclosesand such a characterization may be generated with respect to theuniversal interest space, e.g., defined via concept archive(s) such ascontent taxonomy and/or Wikipedia and/or online encyclopedia, asdiscussed above. For example, each piece of content can be characterizedvia a high dimensional vector with each attribute of the vectorcorresponding to a concept in the interest universe and the value of theattribute indicates whether and/or to what degree the content covers theconcept. When a piece of content is characterized in the same universalinterest space as that for user's profile, the affinity between thecontent and a user profile can be efficiently determined.

Each piece of content in the content pool can also be individuallycharacterized in terms of other criteria. For example, performancerelated measures, such as popularity of the content, may be used todescribe the content. Performance related characterizations of contentmay be used in both selecting content to be incorporated into thecontent pool as well as selecting content already in the content poolfor recommendation of personalized content for specific users. Suchperformance oriented characterizations of each piece of content maychange over time and can be assessed periodically and can be done basedon users' activities. Content pool also changes over time based onvarious reasons, such as content performance, change in users'interests, etc. Dynamically changed performance characterization ofcontent in the content pool may also be evaluated periodically ordynamically based on performance measures of the content so that thecontent pool can be adjusted over time, i.e., by removing lowperformance content pieces, adding new content with good performance, orupdating content.

To grow the content pool, the present teaching discloses ways tocontinually discover both new content and new content sources from whichinteresting content may be accessed, evaluated, and incorporated intothe content pool. New content may be discovered dynamically viaaccessing information from third party applications which users use andexhibit various interests. Examples of such third party applicationsinclude Facebook, Twitter, Microblogs, or YouTube. New content may alsobe added to the content pool when some new interest or an increasedlevel of interests in some subject matter emerges or is predicted basedon the occurrence of certain (spontaneous) events. One example is thecontent about the life of Pope Benedict, which in general may not be atopic of interests to most users but likely will be in light of thesurprising announcement of Pope Benedict's resignation. Such dynamicadjustment to the content pool aims at covering a dynamic (and likelygrowing) range of interests of users, including those that are, e.g.,exhibited by users in different settings or applications or predicted inlight of context information. Such newly discovered content may then beevaluated before it can be selected to be added to the content pool.

Certain content in the content pool, e.g., journals or news, need to beupdated over time. Conventional solutions usually update such contentperiodically based on a fixed schedule. The present teaching disclosesthe scheme of dynamically determining the pace of updating content inthe content pool based on a variety of factors. Content update may beaffected by context information. For example, the frequency at which apiece of content scheduled to be updated may be every 2 hours, but thisfrequency can be dynamically adjusted according to, e.g., an explosiveevent such as an earthquake. As another example, content from a socialgroup on Facebook devoted to Catholicism may normally be updated daily.When Pope Benedict's resignation made the news, the content from thatsocial group may be updated every hour so that interested users can keeptrack of discussions from members of this social group. In addition,whenever there are newly identified content sources, it can be scheduledto update the content pool by, e.g., crawling the content from the newsources, processing the crawled content, evaluating the crawled content,and selecting quality new content to be incorporated into the contentpool. Such a dynamically updated content pool aims at growing incompatible with the dynamically changing users' interests in order tofacilitate quality personalized content recommendation.

Another key to quality personalized content recommendation is the aspectof identifying quality content that meets the interests of a user forrecommendation. Previous solutions often emphasize mere relevance of thecontent to the user when selecting content for recommendation. Inaddition, traditional relevance based content recommendation was mostlybased on short term interests of the user. This not only leads to acontent recommendation bubble, i.e., known short interests causerecommendations limited to the short term interests and reactions tosuch short term interests centric recommendations cycle back to theshort term interests that start the process. This bubble makes itdifficult to come out of the circle to recommend content that can servenot only the overall interests but also long term interests of users.The present teaching combines relevance with performance of the contentso that not only relevant but also quality content can be selected andrecommended to users in a multi-stage ranking system.

In addition, to identify recommended content that can serve a broadrange of interests of a user, the present teaching relies on both shortterm and long term interests of the user to identify user-contentaffinity in order to select content that meets a broader range of users'interests to be recommended to the user.

In content recommendation, monetizing content such as advertisements areusually also selected as part of the recommended content to a user.Traditional approaches often select ads based on content in which theads are to be inserted. Some traditional approaches also rely on userinput such as queries to estimate what ads likely can maximize theeconomic return. These approaches select ads by matching the taxonomy ofthe query or the content retrieved based on the query with the contenttaxonomy of the ads. However, content taxonomy is commonly known not tocorrespond with advertisement taxonomy, which advertisers use to targetat certain audience. As such, selecting ads based on content taxonomydoes not serve to maximize the economic return of the ads to be insertedinto content and recommended to users. The present teaching disclosesmethod and system to build a linkage between content taxonomy andadvertisement taxonomy so that ads that are not only relevant to auser's interests but also the interests of advertisers can be selected.In this way, the recommended content with ads to a user can both servethe user's interests and at the same time to allow the content operatorto enhance monetization via ads.

Yet another aspect of personalized content recommendation of the presentteaching relates to recommending probing content that is identified byextrapolating the currently known user interests. Traditional approachesrely on selecting either random content beyond the currently known userinterests or content that has certain performance such as a high levelof click activities. Random selection of probing content presents a lowpossibility to discover a user's unknown interests. Identifying probingcontent by choosing content for which a higher level of activities areobserved is also problematic because there can be many pieces of contentthat a user may potentially be interested but there is a low level ofactivities associated therewith. The present teaching discloses ways toidentify probing content by extrapolating the currently known interestwith the flexibility of how far removed from the currently knowninterests. This approach also incorporates the mechanism to identifyquality probing content so that there is an enhanced likelihood todiscover a user's unknown interests. The focus of interests at anymoment can be used as an anchor interest based on which probinginterests (which are not known to be interests of the user) can beextrapolated from the anchor interests and probing content can beselected based on the probing interests and recommended to the usertogether with the content of the anchor interests. Probinginterests/content may also be determined based on other considerationssuch as locale, time, or device type. In this way, the disclosedpersonalized content recommendation system can continually explore anddiscover unknown interests of a user to understand better the overallinterests of the user in order to expand the scope of service.

Additional novel features will be set forth in part in the descriptionwhich follows, and in part will become apparent to those skilled in theart upon examination of the following and the accompanying drawings ormay be learned by production or operation of the examples. Theadvantages of the present teachings may be realized and attained bypractice or use of various aspects of the methodologies,instrumentalities and combinations set forth in the detailed examplesdiscussed below.

FIG. 1 depicts an exemplary system diagram 10 for personalized contentrecommendation to a user 105, according to an embodiment of the presentteaching. System 10 comprises a personalized content recommendationmodule 100, which comprises numerous sub modules, content sources 110,knowledge archives 115, third party platforms 120, and advertisers 125with advertisement taxonomy 127 and advertisement database 126. Contentsources 110 may be any source of on-line content such as on-line news,published papers, blogs, on-line tabloids, magazines, audio content,image content, and video content. It may be content from contentprovider such as Yahoo! Finance, Yahoo! Sports, CNN, and ESPN. It may bemulti-media content or text or any other form of content comprised ofwebsite content, social media content, such as Facebook, twitter,Reddit, etc, or any other content rich provider. It may be licensedcontent from providers such AP and Reuters. It may also be contentcrawled and indexed from various sources on the Internet. Contentsources 110 provide a vast array of content to the personalized contentrecommendation module 100 of system 10.

Knowledge archives 115 may be an on-line encyclopedia such as Wikipediaor indexing system such as an on-line dictionary. On-line conceptarchives 115 may be used for its content as well as its categorizationor indexing systems. Knowledge archives 115 provide extensiveclassification system to assist with the classification of both theuser's 105 preferences as well as classification of content. Knowledgeconcept archives, such as Wikipedia may have hundreds of thousands tomillions of classifications and sub-classifications. A classification isused to show the hierarchy of the category. Classifications serve twomain purposes. First they help the system understand how one categoryrelates to another category and second, they help the system maneuverbetween higher levels on the hierarchy without having to move up anddown the subcategories. The categories or classification structure foundin knowledge archives 115 is used for multidimensional content vectorsas well as multidimensional user profile vectors which are utilized bypersonalized content recommendation module 100 to match personalizedcontent to a user 105. Third party platforms 120 maybe any third partyapplications including but not limited to social networking sites likeFacebook, Twitter, LinkedIn, Google+. It may include third party mailservers such as GMail or Bing Search. Third party platforms 120 provideboth a source of content as well as insight into a user's personalpreferences and behaviors.

Advertisers 125 are coupled with the ad content database 126 as well asan ads classification system or ad. taxonomy 127 intended for classifiedadvertisement content. Advertisers 125 may provide streaming content,static content, and sponsored content. Advertising content may be placedat any location on a personalized content page and may be presented bothas part of a content stream as well as a standalone advertisement,placed strategically around or within the content stream.

Personalized content recommendation module 100 comprises applications130, content pool 135, content pool generation/update unit 140,concept/content analyzer 145, content crawler 150, unknown interestexplorer 215, user understanding unit 155, user profiles 160, contenttaxonomy 165, context information analyzer 170, user event analyzer 175,third party interest analyzer 190, social media content sourceidentifier 195, advertisement insertion unit 200 andcontent/advertisement/taxonomy correlator 205. These components areconnected to achieve personalization, content pooling, and recommendingpersonalized content to a user. For example, the content ranking unit210 works in connection with context information analyzer 170, theunknown interest explorer 215, and the ad insertion unit 200 to generatepersonalized content to be recommended to a user with personalized adsor probing content inserted. To achieve personalization, the userunderstanding unit 155 works in connection with a variety of componentsto dynamically and continuously update the user profiles 160, includingcontent taxonomy 165, the knowledge archives 115, user event analyzer175, and the third party interest analyzer 190. Various components areconnected to continuously maintain a content pool, including the contentpool generation/update unit 140, user event analyzer 175, social mediacontent source identifier 195, content/concept analyzer 145, contentcrawler 150, the content taxonomy 165, as well as user profiles 160.

Personalized content recommendation module 100 is triggered when user105 engages with system 10 through applications 130. Applications 130may receive information in the form of a user id, cookies, log ininformation from user 105 via some form of computing device. User 105may access system 10 via a wired or wireless device and may bestationary or mobile. User 105 may interface with the applications 130on a tablet, a Smartphone, a laptop, a desktop or any other computingdevice which may be embedded in devices such as watches, eyeglasses, orvehicles. In addition to receiving insights from the user 105 about whatinformation the user 105 might be interested, applications 130 providesinformation to user 105 in the form of personalized content stream. Userinsights might be user search terms entered to the system, declaredinterests, user clicks on a particular article or subject, user dwelltime or scroll over of particular content, user skips with respect tosome content, etc. User insights may be a user indication of a like, ashare, or a forward action on a social networking site, such asFacebook, or even peripheral activities such as print or scan of certaincontent. All of these user insights or events are utilized by thepersonalized content recommendation module 100 to locate and customizecontent to be presented to user 105. User insights received viaapplications 130 are used to update personalized profiles for userswhich may be stored in user profiles 160. User profiles 160 may bedatabase or a series of databases used to store personalized userinformation on all the users of system 10. User profiles 160 may be aflat or relational database and may be stored in one or more locations.Such user insights may also be used to determine how to dynamicallyupdate the content in the content pool 135.

A specific user event received via applications 130 is passed along touser event analyzer 175, which analyzes the user event information andfeeds the analysis result with event data to the user understanding unit155 and/or the content pool generation/update unit 140. Based on suchuser event information, the user understanding unit 155 estimates shortterm interests of the user and/or infer user's long term interests basedon behaviors exhibited by user 105 over long or repetitive periods. Forexample, a long term interest may be a general interest in sports, whereas a short term interest may be related to a unique sports event, suchas the Super Bowl at a particular time. Over time, a user's long terminterest may be estimated by analyzing repeated user events. A user who,during every engagement with system 10, regularly selects contentrelated to the stock market may be considered as having a long terminterest in finances. In this case, system 10 accordingly, may determinethat personalized content for user 105 should contain content related tofinance. Contrastingly, short term interest may be determined based onuser events which may occur frequently over a short period, but which isnot something the user 105 is interested in the long term. For example,a short term interest may reflect the momentary interest of a user whichmay be triggered by something the user saw in the content but such aninterest may not persist over time. Both short and long term interestare important in terms of identifying content that meets the desire ofthe user 105, but need to be managed separately because of thedifference in their nature as well as how they influence the user.

In some embodiments, short term interests of a user may be analyzed topredict the user's long term interests. To retain a user, it isimportant to understand the user's persistent or long term interests. Byidentifying user 105's short term interest and providing him/her with aquality personalized experience, system 10 may convert an occasionaluser into a long term user. Additionally, short term interest may trendinto long term interest and vice versa. The user understanding unit 155provides the capability of estimating both short and long terminterests.

The user understanding unit 155 gathers user information from multiplesources, including all the user's events, and creates one or moremultidimensional personalization vectors. In some embodiments, the userunderstanding unit 155 receives inferred characteristics about the user105 based on the user events, such as the content he/she views, selfdeclared interests, attributes or characteristics, user activities,and/or events from third party platforms. In an embodiment, the userunderstanding unit 155 receives inputs from social media content sourceidentifier 195. Social media content source identifier 195 relies onuser 105's social media content to personalize the user's profile. Byanalyzing the user's social media pages, likes, shares, etc, socialmedia content source identifier 195 provides information for userunderstanding unit 155. The social media content source identifier 195is capable of recognizing new content sources by identifying, e.g.,quality curators on social media platforms such as Twitter, Facebook, orblogs, and enables the personalized content recommendation module 100 todiscover new content sources from where quality content can be added tothe content pool 135. The information generated by social media contentsource identifier 195 may be sent to a content/concept analyzer 145 andthen mapped to specific category or classification based on contenttaxonomy 165 as well as a knowledge archives 115 classification system.

The third party interest analyzer 190 leverages information from otherthird party platforms about users active on such third party platforms,their interests, as well as content these third party users to enhancethe performance of the user understanding unit 155. For example, wheninformation about a large user population can be accessed from one ormore third party platforms, the user understanding unit 155 can rely ondata about a large population to establish a baseline interest profileto make the estimation of the interests of individual users more preciseand reliable, e.g., by comparing interest data with respect to aparticular user with the baseline interest profile which will capturethe user's interests with a high level of certainty.

When new content is identified from content source 110 or third partyplatforms 120, it is processed and its concepts are analyzed. Theconcepts can be mapped to one or more categories in the content taxonomy165 and the knowledge archives 115. The content taxonomy 165 is anorganized structure of concepts or categories of concepts and it maycontain a few hundred classifications of a few thousand. The knowledgearchives 115 may provide millions of concepts, which may or may not bestructures in a similar manner as the content taxonomy 165. Such contenttaxonomy and knowledge archives may serve as a universal interest space.Concepts estimated from the content can be mapped to a universalinterest space and a high dimensional vector can be constructed for eachpiece of content and used to characterize the content. Similarly, foreach user, a personal interest profile may also be constructed, mappingthe user's interests, characterized as concepts, to the universalinterest space so that a high dimensional vector can be constructed withthe user's interests levels populated in the vector.

Content pool 135 may be a general content pool with content to be usedto serve all users. The content pool 135 may also be structured so thatit may have personalized content pool for each user. In this case,content in the content pool is generated and retained with respect toeach individual user. The content pool may also be organized as a tieredsystem with both the general content pool and personalized individualcontent pools for different users. For example, in each content pool fora user, the content itself may not be physically present but isoperational via links, pointers, or indices which provide references towhere the actual content is stored in the general content pool.

Content pool 135 is dynamically updated by content poolgeneration/update module 140. Content in the content pool comes and goand decisions are made based on the dynamic information of the users,the content itself, as well as other types of information. For example,when the performance of content deteriorates, e.g., low level ofinterests exhibited from users, the content pool generation/update unit140 may decide to purge it from the content pool. When content becomesstale or outdated, it may also be removed from the content pool. Whenthere is a newly detected interest from a user, the content poolgeneration/update unit 140 may fetch new content aligning with the newlydiscovered interests. User events may be an important source of makingobservations as to content performance and user interest dynamics. Useractivities are analyzed by the user event analyzer 175 and suchInformation is sent to the content pool generation/update unit 140. Whenfetching new content, the content pool generation/update unit 140invokes the content crawler 150 to gather new content, which is thenanalyzed by the content/concept analyzer 145, then evaluated by thecontent pool generation/update unit 140 as to its quality andperformance before it is decided whether it will be included in thecontent pool or not. Content may be removed from content pool 135because it is no longer relevant, because other users are notconsidering it to be of high quality or because it is no longer timely.As content is constantly changing and updating content pool 135 isconstantly changing and updating providing user 105 with a potentialsource for high quality, timely personalized content.

In addition to content, personalized content recommendation module 100provides for targeted or personalized advertisement content fromadvertisers 125. Advertisement database 126 houses advertising contentto be inserted into a user's content stream. Advertising content from addatabase 126 is inserted into the content stream via Content rankingunit 210. The personalized selection of advertising content can be basedon the user's profile. Content/advertisement/user taxonomy correlator205 may re-project or map a separate advertisement taxonomy 127 to thetaxonomy associated with the user profiles 160.Content/advertisement/user taxonomy correlator 205 may apply a straightmapping or may apply some intelligent algorithm to the re-projection todetermine which of the users may have a similar or related interestbased on similar or overlapping taxonomy categories.

Content ranking unit 210 generates the content stream to be recommendedto user 105 based on content, selected from content pool 135 based onthe user's profile, as well as advertisement, selected by theadvertisement insertion unit 200. The content to be recommended to theuser 105 may also be determined, by the content ranking unit 210, basedon information from the context information analyzer 170. For example,if a user is currently located in a beach town which differs from thezip code in the user's profile, it can be inferred that the user may beon vacation. In this case, information related to the locale where theuser is currently in may be forwarded from the context informationanalyzer to the Content ranking unit 210 so that it can select contentthat not only fit the user's interests but also is customized to thelocale. Other context information include day, time, and device type.The context information can also include an event detected on the devicethat the user is currently using such as a browsing event of a websitedevoted to fishing. Based on such a detected event, the momentaryinterest of the user may be estimated by the context informationanalyzer 170, which may then direct the Content ranking unit 210 togather content related to fishing amenities in the locale the user is infor recommendation.

The personalized content recommendation module 100 can also beconfigured to allow probing content to be included in the content to berecommended to the user 105, even though the probing content does notrepresent subject matter that matches the current known interests of theuser. Such probing content is selected by the unknown interest explorer215. Once the probing content is incorporated in the content to berecommended to the user, information related to user activities directedto the probing content (including no action) is collected and analyzedby the user event analyzer 175, which subsequently forwards the analysisresult to long/short term interest identifiers 180 and 185. If ananalysis of user activities directed to the probing content reveals thatthe user is or is not interested in the probing content, the userunderstanding unit 155 may then update the user profile associated withthe probed user accordingly. This is how unknown interests may bediscovered. In some embodiments, the probing content is generated basedon the current focus of user interest (e.g., short term) byextrapolating the current focus of interests. In some embodiments, theprobing content can be identified via a random selection from thegeneral content, either from the content pool 135 or from the contentsources 110, so that an additional probing can be performed to discoverunknown interests.

To identify personalized content for recommendation to a user, thecontent ranking unit 210 takes all these inputs and identify contentbased on a comparison between the user profile vector and the contentvector in a multiphase ranking approach. The selection may also befiltered using context information. Advertisement to be inserted as wellas possibly probing content can then be merged with the selectedpersonalized content.

FIG. 2 is a flowchart of an exemplary process for personalized contentrecommendation, according to an embodiment of the present teaching.Content taxonomy is generated at 205. Content is accessed from differentcontent sources and analyzed and classified into different categories,which can be pre-defined. Each category is given some labels and thendifferent categories are organized into some structure, e.g., ahierarchical structure. A content pool is generated at 210. Differentcriteria may be applied when the content pool is created. Examples ofsuch criteria include topics covered by the content in the content pool,the performance of the content in the content pool, etc. Sources fromwhich content can be obtained to populate the content pool includecontent sources 110 or third party platforms 120 such as Facebook,Twitter, blogs, etc. FIG. 3 provides a more detailed exemplary flowchartrelated to content pool creation, according to an embodiment of thepresent teaching. User profiles are generated at 215 based on, e.g.,user information, user activities, identified short/long term interestsof the user, etc. The user profiles may be generated with respect to abaseline population interest profile, established based on, e.g.,information about third party interest, knowledge archives, and contenttaxonomies.

Once the user profiles and the content pool are created, when the system10 detects the presence of a user, at 220, the context information, suchas locale, day, time, may be obtained and analyzed, at 225. FIG. 4illustrates exemplary types of context information. Based on thedetected user's profile, optionally context information, personalizedcontent is identified for recommendation. A high level exemplary flowfor generating personalized content for recommendation is presented inFIG. 5. Such gathered personalized content may be ranked and filtered toachieve a reasonable size as to the amount of content forrecommendation. Optionally (not shown), advertisement as well as probingcontent may also be incorporated in the personalized content. Suchcontent is then recommended to the user at 230.

User reactions or activities with respect to the recommended content aremonitored, at 235, and analyzed at 240. Such events or activitiesinclude clicks, skips, dwell time measured, scroll location and speed,position, time, sharing, forwarding, hovering, motions such as shaking,etc. It is understood that any other events or activities may bemonitored and analyzed. For example, when the user moves the mousecursor over the content, the title or summary of the content may behighlighted or slightly expanded. In another example, when a userinteracts with a touch screen by her/his finger[s], any known touchscreen user gestures may be detected. In still another example, eyetracking on the user device may be another user activity that ispertinent to user behaviors and can be detected. The analysis of suchuser events includes assessment of long term interests of the user andhow such exhibited short term interests may influence the system'sunderstanding of the user's long term interests. Information related tosuch assessment is then forwarded to the user understanding unit 155 toguide how to update, at 255, the user's profile. At the same time, basedon the user's activities, the portion of the recommended content thatthe user showed interests are assessed, at 245, and the result of theassessment is then used to update, at 250, the content pool. Forexample, if the user shows interests on the probing content recommended,it may be appropriate to update the content pool to ensure that contentrelated to the newly discovered interest of the user will be included inthe content pool.

FIG. 3 illustrates different types of context information that may bedetected and utilized in assisting to personalize content to berecommended to a user. In this illustration, context information mayinclude several categories of data, including, but not limited to, time,space, platform, and network conditions. Time related information can betime of the year (e.g., a particular month from which season can beinferred), day of a week, specific time of the day, etc. Suchinformation may provide insights as to what particular set of interestsassociated with a user may be more relevant. To infer the particularinterests of a user at a specific moment may also depend on the localethat the user is in and this can be reflected in the space relatedcontext information, such as which country, what locale (e.g., touristtown), which facility the user is in (e.g., at a grocery store), or eventhe spot the user is standing at the moment (e.g., the user may bestanding in an aisle of a grocery store where cereal is on display).Other types of context information includes the specific platformrelated to the user's device, e.g., Smartphone, Tablet, laptop, desktop,bandwidth/data rate allowed on the user's device, which will impact whattypes of content may be effectively presented to the user. In addition,the network related information such as state of the network where theuser's device is connected to, the available bandwidth under thatcondition, etc. may also impact what content should be recommended tothe user so that the user can receive or view the recommended contentwith reasonable quality.

FIG. 4 depicts an exemplary system diagram of the content poolgeneration/update unit 140, according to an embodiment of the presentteaching. The content pool 135 can be initially generated and thenmaintained according to the dynamics of the users, contents, and needsdetected. In this illustration, the content pool generation/update unit140 comprises a content/concept analyzing control unit 410, a contentperformance estimator 420, a content quality evaluation unit 430, acontent selection unit 480, which will select appropriate content toplace into the content pool 135. In addition, to control how content isto be updated, the content pool generation/update unit 140 also includesa user activity analyzer 440, a content status evaluation unit 450, anda content update control unit 490.

The content/concept analyzing control unit 410 interfaces with thecontent crawler 150 (FIG. 1) to obtain candidate content that is to beanalyzed to determine whether the new content is to be added to thecontent pool. The content/concept analyzing control unit 410 alsointerfaces with the content/concept analyzer 145 (see FIG. 1) to get thecontent analyzed to extract concepts or subjects covered by the content.Based on the analysis of the new content, a high dimensional vector forthe content profile can be computed via, e.g., by mapping the conceptsextracted from the content to the universal interest space, e.g.,defined via Wikipedia or other content taxonomies. Such a contentprofile vector can be compared with user profiles 160 to determinewhether the content is of interest to users. In addition, content isalso evaluated in terms of its performance by the content performanceestimator 420 based on, e.g., third party information such as activitiesof users from third party platforms so that the new content, althoughnot yet acted upon by users of the system, can be assessed as to itsperformance. The content performance information may be stored, togetherwith the content's high dimensional vector related to the subject of thecontent, in the content profile 470. The performance assessment is alsosent to the content quality evaluation unit 430, which, e.g., will rankthe content in a manner consistent with other pieces of content in thecontent pool. Based on such rankings, the content selection unit 480then determines whether the new content is to be incorporated into thecontent pool 135.

To dynamically update the content pool 135, the content poolgeneration/update unit 140 may keep a content log 460 with respect toall content presently in the content pool and dynamically update the logwhen more information related to the performance of the content isreceived. When the user activity analyzer 440 receives informationrelated to user events, it may log such events in the content log 460and perform analysis to estimate, e.g., any change to the performance orpopularity of the relevant content over time. The result from the useractivity analyzer 440 may also be utilized to update the contentprofiles, e.g., when there is a change in performance. The contentstatus evaluation unit 450 monitors the content log and the contentprofile 470 to dynamically determine how each piece of content in thecontent pool 135 is to be updated. Depending on the status with respectto a piece of content, the content status evaluation unit 450 may decideto purge the content if its performance degrades below a certain level.It may also decide to purge a piece of content when the overall interestlevel of users of the system drops below a certain level. For contentthat requires update, e.g., news or journals, the content statusevaluation unit 450 may also control the frequency 455 of the updatesbase d on the dynamic information it receives. The content updatecontrol unit 490 carries out the update jobs based on decisions from thecontent status evaluation unit 450 and the frequency at which certaincontent needs to be updated. The content update control unit 490 mayalso determine to add new content whenever there is peripheralinformation indicating the needs, e.g., there is an explosive event andthe content in the content pool on that subject matter is not adequate.In this case, the content update control unit 490 analyzes theperipheral information and if new content is needed, it then sends acontrol signal to the content/concept analyzing control unit 410 so thatit can interface with the content crawler 150 to obtain new content.

FIG. 5 is a flowchart of an exemplary process of creating the contentpool, according to an embodiment of the present teaching. Content isaccessed at 510 from content sources, which include content from contentportals such as Yahoo!, general Internet sources such as web sites orFTP sites, social media platforms such as Twitter, or other third partyplatforms such as Facebook. Such accessed content is evaluated, at 520,as to various considerations such as performance, subject matterscovered by the content, and how it fit users' interests. Based on suchevaluation, certain content is selected to generate, at 530, the contentpool 135, which can be for the general population of the system or canalso be further structured to create sub content pools, each of whichmay be designated to a particular user according to the user'sparticular interests. At 540, it is determined whether user-specificcontent pools are to be created. If not, the general content pool 135 isorganized (e.g., indexed or categorized) at 580. If individual contentpools for individual users are to be created, user profiles are obtainedat 550, and with respect to each user profile, a set of personalizedcontent is selected at 560 that is then used to create a sub contentpool for each such user at 570. The overall content pool and the subcontent pools are then organized at 580.

FIG. 6 is a flowchart of an exemplary process for updating the contentpool 135, according to an embodiment of the present teaching. Dynamicinformation is received at 610 and such information includes useractivities, peripheral information, user related information, etc. Basedon the received dynamic information, the content log is updated at 620and the dynamic information is analyzed at 630. Based on the analysis ofthe received dynamic information, it is evaluated, at 640, with respectto the content implicated by the dynamic information, as to the changeof status of the content. For example, if received information isrelated to user activities directed to specific content pieces, theperformance of the content piece may need to be updated to generate anew status of the content piece. It is then determined, at 650, whetheran update is needed. For instance, if the dynamic information from aperipheral source indicates that content of certain topic may have ahigh demand in the near future, it may be determined that new content onthat topic may be fetched and added to the content pool. In this case,at 660, content that needs to be added is determined. In addition, ifthe performance or popularity of a content piece has just dropped belowan acceptable level, the content piece may need to be purged from thecontent pool 135. Content to be purged is selected at 670. Furthermore,when update is needed for regularly refreshed content such as journal ornews, the schedule according to which update is made may also be changedif the dynamic information received indicates so. This is achieved at680.

FIG. 7 depicts an exemplary diagram of the user understanding unit 155,according to an embodiment of the present teaching. In this exemplaryconstruct, the user understanding unit 155 comprises a baseline interestprofile generator 710, a user profile generator 720, a userintent/interest estimator 740, a short term interest identifier 750 anda long term interest identifier 760. In operation, the userunderstanding unit 155 takes various input and generates user profiles160 as output. Its input includes third party data such as users'information from such third party platforms as well as content suchusers accessed and expressed interests, concepts covered in such thirdparty data, concepts from the universal interest space (e.g., Wikipediaor content taxonomy), information about users for whom the personalizedprofiles are to be constructed, as well as information related to theactivities of such users. Information from a user for whom apersonalized profile is to be generated and updated includesdemographics of the user, declared interests of the user, etc.Information related to user events includes the time, day, location atwhich a user conducted certain activities such as clicking on a contentpiece, long dwell time on a content piece, forwarding a content piece toa friend, etc.

In operation, the baseline interest profile generator 710 accessinformation about a large user population including users' interests andcontent they are interested in from one or more third party sources(e.g., Facebook). Content from such sources is analyzed by thecontent/concept analyzer 145 (FIG. 1), which identifies the conceptsfrom such content. When such concepts are received by the baselineinterest profile generator 710, it maps such concepts to the knowledgearchives 115 and content taxonomy 165 (FIG. 1) and generate one or morehigh dimensional vectors which represent the baseline interest profileof the user population. Such generated baseline interest profile isstored at 730 in the user understanding unit 155. When there is similardata from additional third party sources, the baseline interest profile730 may be dynamically updated to reflect the baseline interest level ofthe growing population.

Once the baseline interest profile is established, when the user profilegenerator receives user information or information related to estimatedshort term and long term interests of the same user, it may then map theuser's interests to the concepts defined by, e.g., the knowledgearchives or content taxonomy, so that the user's interests are nowmapped to the same space as the space in which the baseline interestprofile is constructed. The user profile generator 720 then compares theuser's interest level with respect to each concept with that of a largeruser population represented by the baseline interest profile 730 todetermine the level of interest of the user with respect to each conceptin the universal interest space. This yields a high dimensional vectorfor each user. In combination with other additional information, such asuser demographics, etc., a user profile can be generated and stored in160.

User profiles 160 are updated continuously based on newly receiveddynamic information. For example, a user may declare additionalinterests and such information, when received by the user profilegenerator 720, may be used to update the corresponding user profile. Inaddition, the user may be active in different applications and suchactivities may be observed and information related to them may begathered to determine how they impact the existing user profile and whenneeded, the user profile can be updated based on such new information.For instance, events related to each user may be collected and receivedby the user intent/interest estimator 740. Such events include that theuser dwelled on some content of certain topic frequently, that the userrecently went to a beach town for surfing competition, or that the userrecently participated in discussions on gun control, etc. Suchinformation can be analyzed to infer the user intent/interests. When theuser activities relate to reaction to content when the user is online,such information may be used by the short term interest identifier 750to determine the user's short term interests. Similarly, someinformation may be relevant to the user's long term interests. Forexample, the number of requests from the user to search for contentrelated to diet information may provide the basis to infer that the useris interested in content related to diet. In some situations, estimatinglong term interest may be done by observing the frequency and regularityat which the user accesses certain type of information. For instance, ifthe user repeatedly and regularly accesses content related to certaintopic, e.g., stocks, such repetitive and regular activities of the usermay be used to infer his/her long term interests. The short terminterest identifier 750 may work in connection with the long terminterest identifier 760 to use observed short term interests to inferlong term interests. Such estimated short/long term interests are alsosent to the user profile generator 720 so that the personalization canbe adapted to the changing dynamics.

FIG. 8 is a flowchart of an exemplary process for generating a baselineinterest profile based on information related to a large userpopulation, according to an embodiment of the present teaching. Thethird party information, including both user interest information aswell as their interested content, is accessed at 810 and 820. Thecontent related to the third party user interests is analyzed at 830 andthe concepts from such content are mapped, at 840 and 850, to knowledgearchives and/or content taxonomy. To build a baseline interest profile,the mapped vectors for third party users are then summarized to generatea baseline interest profile for the population. There can be a varietyways to summarize the vectors to generate an averaged interest profilewith respect to the underlying population.

FIG. 9 is a flowchart of an exemplary process for generating/updating auser profile, according to an embodiment of the present teaching. Userinformation is received first at 910. Such user information includesuser demographics, user declared interests, etc. Information related touser activities is also received at 920. Content pieces that are knownto be interested by the user are accessed at 930, which are thenanalyzed, at 950, to extract concepts covered by the content pieces. Theextracted concepts are then mapped, at 960, to the universal interestspace and compared with, concept by concept, the baseline interestprofile to determine, at 970, the specific level of interest of the usergiven the population. In addition, the level of interests of each usermay also be identified based on known or estimated short and long terminterests that are estimated, at 940 and 945, respectively, based onuser activities or content known to be interested by the user. Apersonalized user profile can then be generated, at 980, based on theinterest level with respect to each concept in the universal interestspace.

FIG. 10 depicts an exemplary system diagram for the content ranking unit210, according to an embodiment of the present teaching. The contentranking unit 210 takes variety of input and generates personalizedcontent to be recommended to a user. The input to the content rankingunit 210 includes user information from the applications 130 with whicha user is interfacing, user profiles 160, context informationsurrounding the user at the time, content from the content pool 135,advertisement selected by the ad insertion unit 200, and optionallyprobing content from the unknown interest explorer 215. The contentranking unit 210 comprises a candidate content retriever 1010 and amulti-phase content ranking unit 1020. Based on user information fromapplications 130 and the relevant user profile, the candidate contentretriever 1010 determines the content pieces to be retrieved from thecontent pool 135. Such candidate content may be determined in a mannerthat is consistent with the user's interests or individualized. Ingeneral, there may be a large set of candidate content and it needs tobe further determined which content pieces in this set are mostappropriate given the context information. The multi-phase contentranking unit 1020 takes the candidate content from the candidate contentretriever 1010, the advertisement, and optionally may be the probingcontent, as a pool of content for recommendation and then performsmultiple stages of ranking, e.g., relevance based ranking, performancebased ranking, etc. as well as factors related to the contextsurrounding this recommendation process, and selects a subset of thecontent to be presented as the personalized content to be recommended tothe user.

FIG. 11 is a flowchart of an exemplary process for the content rankingunit, according to an embodiment of the present teaching. User relatedinformation and user profile are received first at 1110. Based on thereceived information, user's interests are determined at 1120, which canthen be used to retrieve, at 1150, candidate content from the contentpool 135. The user's interests may also be utilized in retrievingadvertisement and/or probing content at 1140 and 1130, respectively.Such retrieved content is to be further ranked, at 1160, in order toselect a subset as the most appropriate for the user. As discussedabove, the selection takes place in a multi-phase ranking process, eachof the phases is directed to some or a combination of ranking criteriato yield a subset of content that is not only relevant to the user as tointerests but also high quality content that likely will be interestedby the user. The selected subset of content may also be furtherfiltered, at 1170, based on, e.g., context information. For example,even though a user is in general interested in content about politicsand art, if the user is currently in Milan, Italy, it is likely that theuser is on vacation. In this context, rather than choosing contentrelated to politics, the content related to art museums in Milan may bemore relevant. The multi-phase content ranking unit 1020 in this casemay filter out the content related to politics based on this contextualinformation. This yields a final set of personalized content for theuser. At 1180, based on the contextual information associated with thesurrounding of the user (e.g., device used, network bandwidth, etc.),the content ranking unit packages the selected personalized content, at1180, in accordance with the context information and then transmits, at1190, the personalized content to the user.

More detailed disclosures of various aspects of the system 10,particularly the personalized content recommendation module 100, arecovered in different U.S. patent applications as well as PCTapplications, entitled “Method and System For User Profiling Via MappingThird Party Interests To A Universal Interest Space”, “Method and Systemfor Multi-Phase Ranking For Content Personalization”, “Method and Systemfor Measuring User Engagement Using Click/Skip In Content Stream”,“Method and System for Dynamic Discovery And Adaptive Crawling ofContent From the Internet”, “Method and System For Dynamic Discovery ofInteresting URLs From Social Media Data Stream”, “Method and System forDiscovery of User Unknown Interests”, “Method and System for EfficientMatching of User Profiles with Audience Segments”, “Method and SystemFor Mapping Short Term Ranking Optimization Objective to Long TermEngagement”, “Social Media Based Content Selection System”, “Method andSystem For Measuring User Engagement From Stream Depth”, “Method andSystem For Measuring User Engagement Using Scroll Dwell Time”, “AlmostOnline Large Scale Collaborative Based Recommendation System”, and“Efficient and Fault-Tolerant Distributed Algorithm for Learning LatentFactor Models through Matrix Factorization”. The present teaching isparticularly directed to measuring user engagement in personalizedcontent recommendation.

One of the major challenges in personalized content recommendation is tofind the good signals or representations of user interests andengagement. Traditionally, user engagement in personalized contentrecommendation is measured based on explicit user activities/actions,such as clicking, or other definitive interactions. For example, given aset of recommended content on the web page, the content that has beenexplicitly interacted with is considered as being engaged by the usersand thus, is assigned a positive label for engagement, whereaseverything else on the web page is considered as not being engaged andthus, is assigned a negative label.

This assumption, however, may not be true, in particular, inrecommending “endless” streams of information, in which a user cancontinue to scroll down the page and new content is continually loaded.Such manner of recommending content in personalized content streams hasbecome more and more popular on mobile platforms and also some desktopapplications. More websites and applications are shifting theirdirections of display content to an infinite stream format rather than apaginated form. However, existing systems still measure engagement basedon clicking or other explicit interactions, although content streams inthese system may often be consumed without the users providing anexplicit action. In such cases, the user is engaged with the system, buttheir engagement is difficult to measure.

It is not sufficient to define and measure engagement based solely onexplicit or definitive interactions, such as clicking, because userssometimes may prefer to browse the available visible information in acontent stream without explicitly clicking to view full content ordetails, which is particularly common on mobile and tablet platforms dueto the limited screen size. In other words, measuring engagements solelybased on definitive interactions does not account for use patternsinvolving passive browsing. For example, users who browse contentwithout clicking on it would be considered “not engaged” according toexplicit action-based metrics, despite the fact that they may spendsignificant time and/or browse deeply into the content stream, in eithera desktop or mobile interface. In addition, content that can be consumedwithout clicking on it (e.g., a news article title with abstract that isread but not clicked through to the full article) cannot have theirlevels of engagement measured. Moreover, it is also a common browsingparadigm in Asian markets like China, Korea and Japan, where theinformation density of the languages leads to a more passive streambrowsing patterns for users. Therefore, there is a need to provideimproved solutions for measuring user engagement in personalized contentrecommendation to solve the above-mentioned problems.

The present teaching describes methods, systems, and programming aspectsof measuring user engagement in personalized content recommendation. Thepresent teaching describes novel user engagement metrics, includingclick odds, skip odds, abandon odds, stream depth, and scroll dwelltime, for measuring user engagement with a personalized contentrecommendation system. The engagement metrics described in the presentteaching are better indicators of user's both explicit interactions andimplicit interactions with continuous content streams compared with thetraditional metrics, such as CTR. The methods and system as describedherein are capable of driving more accurate signals from userinteractions with content stream, even with a passive browsing pattern,which in turn improves the recommendation quality and drives userengagement. The methods and systems as described herein allowcalculating user engagement scores with respect to each piece of contentusing one or more novel user engagement metrics as described herein,which then may be used as a basis for inferring user interests andbuilding user profiles for user understanding and/or selecting andranking content for content recommendation.

In one aspect of the present teaching, user engagement is modeled byconsidering skipping in conjunction with clicking and session/pageabandoning to obtain unbiased estimates of content popularity. Themethod and system in this aspect of present teaching consider varioustypes of user action jointly, including explicit actions such asclicking and implicit actions such as skipping and abandoning, todetermine the quality of content. Preliminary experiment results show amore than 5% increase of user engagement after switching rankingoptimization target from CTR to the novel metrics described in thisaspect the present teaching.

In another aspect of the present teaching, the method and system defineand measure engagement without the use of clicking. In this aspect, userengagement measure is based on how far the user scrolls through thecontent stream before abandoning the stream or page. The method andsystem in this aspect score individual content based on position andstream depth without relying on explicit actions like clicking. Usingstream depth is useful particularly when there is an infinite stream ofcontent, which is popular on mobile devices and some desktop interfaces,and when passively browsing content is a common use pattern.

In still another aspect of the present teaching, the method and systemdefine and measure engagement without the use of clicks. In this aspect,user engagement measure is based on the user's dwell time on the contentstream when the user scrolls through the content before abandoning thestream or page, i.e., scroll dwell time. This aspect of the presentteaching is available for any web-based personalization system, and cando web-scale personalization where most of the users do not login forprivacy concern. The method and system in this aspect of the presentteaching do not require users to click, thereby imposing no cost on theuser.

FIG. 12 is a high level exemplary system diagram of a user engagementassessment system, according to an embodiment of the present teaching.The user engagement assessment system 1200 is configured to measure userengagement in personalized content recommendation based on useractivities with respect to personalized content stream. The contentreferred herein includes, but is not limited to, for example, text,audio, image, video or any combination thereof. The user engagementassessment system 1200 may calculate user engagement scores 1201(utility values) of one or more metrics, such as but not limited to,click odds, skip odds, abandon odds, stream depths, and stroll dwelltimes, with respect to each piece of content in a content stream andprovide the user engagement scores 1201 to a personalized contentrecommendation system 1202. The user engagement scores 1201 may be usedby the personalized content recommendation system 1202 for building userprofiles and predicting user interests, selecting and updating content,and/or optimizing ranking model for content recommendation to users1204. The user engagement assessment system 1200 in FIG. 12 may work asthe user event analyzer 175 in FIG. 1 or achieve some functions of theuser event analyzer 175 as described before. The personalized contentrecommendation system 1202 in FIG. 12 may be, for example, thepersonalized content recommendation module 100 in FIG. 1 or any othersystem, module, or unit for content personalization and recommendation.

In this example, the user engagement assessment system 1200 includes auser activity detection module 1206 for detecting different useractivities with respect to personalized content stream using varioustechniques. The user activities may include explicit actions, such asclicking a piece of content, and implicit actions, such as viewing,skipping one or more pieces of content in a content stream, scrollingthrough a content stream, or abandoning a content stream or a page. Thetechniques used for detecting user activities may include, for example,online monitoring by web beacons (web bugs) or tool bars and offlineanalysis of event logs using browser-cookies.

Referring now to FIGS. 23-25, exemplary user activities with respect toa content stream are illustrated. In FIGS. 23-25, content in a contentstream is continuously displayed on a web page 2302 for a user via auser device 2300. In FIG. 23, each piece of content 2304, 2306, 2308 maybe arranged in an order from top to bottom of the web page 2302.Depending on the screen size and/or display resolution of the userdevice 2300, a certain number of content items may be displayed on theweb page 2302 simultaneously. Each piece of content may be associatedwith certain properties, such as the position (ranking) in the contentstream and the presentation style. For example, the content 2304 has aposition 1 as it is the first piece of content in the stream and apresentation style of “pure text”; the content 2306 has a position of 2and a presentation style of “text plus small thumbnail”; the content2308 has a position of 3 and a presentation style of “text plus largethumbnail.” It is understood the position indicates the sequence ofpresenting each content item in the stream and thus, may not alwaysincrement from top to bottom if the content stream flows in a differentdirection, e.g., from left to right or bottom to top. It is alsounderstood that the presentation styles are not limited to theabove-mentioned examples and may be text, image, video, animation,audio, or any combination thereof. In FIG. 23, due to the limited screensize, the user needs to scroll down or up the content stream in order toview more content. The action of “scrolling” 2310 described in thepresent teaching may include any user actions/activities that can causecontent in a content stream to be continuously presented. Scrolling maybe triggered by touch screen gestures or computer mouse motion or a keypress and continue without further intervention until a further useraction, or be entirely controlled by input devices.

Moving to FIG. 24, the user's scrolling action causes new content to bedisplayed on the display screen, replacing the previous content 2304,2306, 2308 shown in FIG. 23. The user may find a particular content 2402interesting and want to explore the content 2402 by applying a“clicking” action on it. The action of “clicking” 2404 described in thepresent teaching may include any explicit actions/activities thatindicate selection of a piece of content in the stream, such as but notlimited to, mouse clicking, pressing/touching on touch screen, keypressing, etc. On the other hand, once a content item is clicked, allthe un-clicked content items above it in the stream are considered being“skipped.” Accordingly, the action of “skipping” may be introduced as animplicit user action/activity with respect to content streams. In FIG.24, the explicit action of clicking content 2402 brings implicit actionsof skipping content 2406, 2408, 2410.

Moving to FIG. 25, after checking out the details of content 2402 inFIG. 24, the user may continue to scroll down the content stream andmore new content items are presented. For some reasons, when the userscrolls to content 2502, she/or he decides to abandon browsing thestream/session by, for example, closing the entire web page 2502 or anapplication that displays the content stream, and clicking on nothing inthe content stream. The action of “abandoning” 2504 described in thepresent teaching may include any user actions/activities that stop acontent stream. FIGS. 23-25 illustrate several user activities, e.g.,clicking, scrolling, skipping, and abandoning, with respect to contentin a content stream, which are monitored by the user activity detectionmodule 1206.

Referring back to FIG. 12, the user engagement assessment system 1200may also include a user engagement evaluation module 1208 for analyzingdetected user activities and scoring individual content using one ormore novel metrics as indicators of degree of engagement. Depending onthe specific metric that is used, different types of user activities orcombination of user activities may be measured and analyzed usingdifferent models and functions as will be described later in detail.

In this example, the personalized content recommendation system 1202includes a ranking model optimization module 1210, a content poolgeneration/update module 1212, and a user understanding module 1214. Theuser engagement scores 1201 for each piece of content may be used by anyor all of these modules 1210, 1212, 1214 for different purposes. In oneexample, the user engagement scores 1201 may be used as machine learningtargets by the ranking model optimization module 1210 to optimize theranking model 1216 for content ranking. In another example, the userengagement scores 1201 collected from a number of users may be used as aquality or popularity indicator by the content pool generation/updatemodule 1212 to select content in order to build and update a contentpool 1218. In still another example, the user engagement scores 1201 fora specific user may be used as a relevancy indicator by the userunderstanding module 1214 to infer user interests in order to build andupdate the user's profile 1220. The user profiles 1220, content pool1218, and ranking model 1216 are all essential components for makingaccurate personalized content recommendation to the users 1204 made by acontent ranking module 1222.

FIG. 13 depicts exemplary applications of user engagement measures,according to an embodiment of the present teaching. In one embodiment,from the content's perspective, user engagement measures/scores E1, E2,. . . , En with respect to user 1, user 2, . . . , user n in a usercohort may be used to estimate popularity and/or quality of each pieceof content in the user cohort. In one example, content having a highclick odds and/or low skip odds and abandon odds with respect to a groupof users indicate its high popularity among those users. In anotherexample, content that is always right above where users stop the contentstream (stream depth) usually has a low quality. In still anotherexample, content on which quite a few users stay longer when scrollingthrough the content streams is considered to be popular among thoseusers and have a high quality.

In another embodiment, from the users' perspective, user engagementmeasures/scores E1′, E2′, . . . , En′ with respect to different piecesof content may be used to predict each user's interest. For each pieceof content, the value of user engagement measures/scores indicates theuser's degree of interest in the content, i.e., a degree of relevancybetween the user and the content. In one example, assuming a user stopsa content stream after reading 20 articles, it strongly suggests thatthe user is interested in the first several articles otherwise she/hewould have stopped earlier. In another example, quickly scrolling downthe first several articles in a content stream, i.e., a short strolldwell time, shows a user's lack of interest in these articles. Bycollecting the same user's engagement measures/scores E1′, E2′, . . . ,En′ with respect to different pieces of content and analyzing thefeatures/topics of the content, the specific user's interest may also beinferred. For example, if a user's engagement measures/scores are alwayshigh for certain articles with the same topic, it is reasonable tosuggest that the user is likely interested in this topic.

FIG. 14 is a function block diagram of one example of the userengagement assessment system shown in FIG. 12, according to anembodiment of the present teaching. In this embodiment, a userengagement assessment system 1400 is capable of estimating unbiasedpopularity from user activities with respect to a personalized contentstream 1402. The user engagement assessment system 1400 in thisembodiment captures how a particular piece of content deviates from mostother content in terms of either increase or decrease engagement withthe personalized content stream 1402. To measure such deviation, theuser engagement assessment system 1400 calculates user engagementscores, e.g., odds ratio including click odds, skip odds, and abandonodds, with respect to each piece of content. In this embodiment, theuser engagement assessment system 1400 includes a user activitydetection module 1404 for detecting user activities, including clicking,skipping, and abandoning actions, and a user engagement evaluationmodule 1406 for computing the user engagement scores, including clickodds, skip odds, and abandon odds.

In this embodiment, the user activity detection module 1404 includes aclicking action detection unit 1408, a skipping action detection unit1410, and an abandoning action detection unit 1412, which are configuredto detect clicking, skipping and abandoning actions, respectively. Theclicking, skipping and abandoning actions with respect to thepersonalized content stream 1402 have been described before with respectto FIGS. 23-25. In this embodiment, the clicking action is assumed to bea vote of relevancy from the users and is a positive signal, whereasskipping and abandoning actions imply that the uses find the contentless or totally irrelevant and thus are negative signals. The detectionsof the user activities may be performed over a detection period 1414 forgeneral users or a user cohort 1416, i.e., a group of users determinedbased on, for example, demographics, visit patterns, on page behaviors,or any other user profile. It is understood that the detections may bedone over an extended time period for creating event logs for offlineanalysis or done in near real-time for computing the user engagementscores for a particular piece of content.

In this embodiment, the user engagement evaluation module 1406 includesa user activity log database 1418, a model building unit 1420, a userengagement score calculation unit 1422, and a content bias propertyextraction unit 1424. The user activity log database 1418 in thisembodiment stores user events, including clicking, skipping, andabandoning actions, detected over the detection period 1414, whichcovers a significant user base and content pool. The model building unit1420 is responsible for building a probability model 1426 for averagecontent based on the data from the user activity log database 1418. Thedetails of building the probability model 1426 will be described laterin FIG. 15. The user engagement score calculation unit 1422 thenreceives data of user activities, including clicking, skipping, andabandoning actions, in near real-time and calculates click odds, skipodds, and abandon odds, respectively, based on the probability model1426.

It is understood that some properties of the content, such as theposition in the stream and presentation style (format), may introducebias in user engagement measurement. For example, an article presentedwith a big thumbnail may be visually attractive for a user to click eventhough the user may not be really interested in the topic of thatarticle. In some examples, the bias caused by the content biasproperties needs to be considered by the model building unit 1420 inbuilding the probability model 1426 and/or by the user engagement scorecalculation unit 1422 in computing the user engagement scores. Thecontent bias property extraction unit 1424 is configured to extractthose properties for each piece of content such that the impact thereofmay be captured and eliminated in order to measure an unbiasedpopularity of the content.

FIGS. 15-16 are flowcharts of an exemplary process of the userengagement assessment system shown in FIG. 14, according to differentembodiments of the present teaching. FIG. 15 illustrates one example ofbuilding the probability model 1426 in FIG. 14. Starting from 1502, atarget user cohort is first determined. That is, the probability modelmay be build for a specific type of users based on their demographics,visit patterns, on page behaviors or any other user profiles. It isunderstood that 1502 may not be necessary if the probability model isbuilt for general users, and the target users could be all the users whohave interacted with the content stream in the detection period or userswho have been randomly picked up. At 1504, a detection period is alsodetermined. In one example, the detection period may be an extendedperiod of time, e.g., one month, which covers a significant user baseand content pool.

Moving to 1506, user activities including clicking, skipping andabandoning actions are monitored during the detection period for theuser cohort or general users. As described before, content biasproperties such as position and presentation style may be also extractedand recorded. In one example, each content item d is associated with aposition i, which is a non-negative integer denoting the item's rank inthe stream. For example, the top item in the stream is in position 1,and the slot immediately below is position 2, etc. Each content item isalso presented using one of the predefined presentation styles (formats)indicated by j. For example, the presentation styles may include (1)pure text only, (2) text plus small thumbnail, and (3) text plus bigthumbnail. Upon examining a content item d at position i in presentationstyle j, a user may perform one of the three following actions: (1)click on d, (2) skip d and start examining the following article, and(3) abandon the session (i.e. click on nothing in the stream). Thenumbers of occurrence of these three types of events involving item d atposition i in presentation style j may be represented as C_(dij),S_(dij), and A_(dij), respectively. It is understood that the clicking,skipping and abandoning actions referred herein are not limited tospecific user actions and may include various user actions that maycause the same or similar effect as “clicking,” “skipping,” and“abandoning” in different applications, user interfaces, operatingsystem and/or user devices. For example, in some user interfaces orapplications, hovering over a piece of content or gazing on the contentover a certain period of time may have the same effect as clicking onthe content, i.e., selecting the content, and thus, are also consideredas a clicking action in the present teaching.

At 1508, the monitored data, i.e., event logs from browser-cookies maybe consolidated and analyzed to compute probabilities of clicking,skipping and abandoning actions. The probabilities may be calculated asspecific numbers, or parameters that need to be estimated. In theabove-mentioned example, using the event logs over an extended period oftime, which covers a significant user base and content pool, thefollowing probabilities may be computed:

$\begin{matrix}{{P\left( {{{click}i},j} \right)} = \frac{\begin{matrix}{\# \; {times}\mspace{14mu} {any}\mspace{14mu} {items}\mspace{14mu} {got}\mspace{14mu} {clicked}} \\{{at}\mspace{14mu} {position}\mspace{14mu} i\mspace{14mu} {in}\mspace{14mu} {format}\mspace{14mu} j}\end{matrix}}{\begin{matrix}{\# \; {times}\mspace{14mu} {any}\mspace{14mu} {items}\mspace{14mu} {got}\mspace{14mu} {presented}} \\{{at}\mspace{14mu} {position}\mspace{14mu} i\mspace{14mu} {in}\mspace{14mu} {format}\mspace{14mu} j}\end{matrix}}} & (1) \\{{P\left( {{{skip}i},j} \right)} = \frac{\begin{matrix}{\# \; {times}\mspace{14mu} {any}\mspace{14mu} {items}\mspace{14mu} {got}\mspace{14mu} {skipped}} \\{{at}\mspace{14mu} {position}\mspace{14mu} i\mspace{14mu} {in}\mspace{14mu} {format}\mspace{14mu} j}\end{matrix}}{\begin{matrix}{\# \; {times}\mspace{14mu} {any}\mspace{14mu} {items}\mspace{14mu} {got}\mspace{14mu} {presented}} \\{{at}\mspace{14mu} {position}\mspace{14mu} i\mspace{14mu} {in}\mspace{14mu} {format}\mspace{14mu} j}\end{matrix}}} & (2) \\{{P\left( {{{abandon}i},j} \right)} = {\frac{\begin{matrix}{\# \; {times}\mspace{14mu} {any}\mspace{14mu} {items}\mspace{14mu} {got}\mspace{14mu} {abandoned}} \\{{at}\mspace{14mu} {position}\mspace{14mu} i\mspace{14mu} {in}\mspace{14mu} {format}\mspace{14mu} j}\end{matrix}}{\begin{matrix}{\# \; {times}\mspace{14mu} {any}\mspace{14mu} {items}\mspace{14mu} {got}\mspace{14mu} {presented}} \\{{at}\mspace{14mu} {position}\mspace{14mu} i\mspace{14mu} {in}\mspace{14mu} {format}\mspace{14mu} j}\end{matrix}}.}} & (3)\end{matrix}$

At 1510, the probabilities model is built based on the probabilities. Inthe above-mentioned example, the three probabilities may be referred toas a bias probability model, as it captures the impact of positions andpresentation styles on user behaviors regardless of the items. It isunderstood that variations of the probabilities model described inEquations 1-3 may be made in some examples. In one example, in additionto position and presentation style (format), time dependency may also beincluded as another bias such that each event may be weighted to takeinto account of position, presentation style (format), and/or timedependency. In another example, a prior may be added in the numeratorand denominator in Equations 1-3 to smooth out cases with low counts.

FIG. 16 illustrates one example of calculating user engagement scoreswith respect to each piece of content based on the probability model1426 in FIG. 14. Starting from 1602, a target user cohort is firstdetermined. The user engagement scores may be designated to a specificuser cohort or even a specific user if desired. It is understood that1602 may not be necessary if the user engagement scores are for generalusers. At 1604, a detection period is also determined. In one example,the detection period may be shorter than the detection period forbuilding the probability model in FIG. 15. For example, the detectionperiod in 1604 may be one day. Moving to 1606, a target content item isdetermined. As the user engagement scores are with respect to eachspecific piece of content, the target content may be determined first.

At 1608, user activities including clicking, skipping and abandoningactions are monitored during the detection period for the user cohort orgeneral users. As described before, content bias properties, such asposition and presentation style, may be also extracted and recorded. At1610, the actual numbers of occurrence of each of the clicking, skippingand abandoning actions are counted, and the probability model isretrieved at 1612. Moving to 1614, each expected number of occurrence ofclicking, skipping, and abandoning actions is estimated, for example,based on the total number of occurrences of the three types of actionsand the probability model. Eventually, at 1616, user engagement scoresfor the target content are calculated based on the actual occurrences ofeach of the clicking, skipping and abandoning actions and theircorresponding expected occurrences.

Referring to the example mentioned in FIG. 15, the user engagementscores are odds ratio including click odds (γ_(d,click)), skip odds(γ_(d,skip)), and abandon odds (γ_(d,abandon)), each of which isrespectively computed as following:

$\begin{matrix}{{\gamma_{d,{click}} = \frac{\sum_{i,j}C_{dij}}{\sum_{i,j}{{P\left( {{{click}i},j} \right)}\left( {C_{dij} + S_{dij} + A_{dij}} \right)}}}\mspace{14mu}} & (4) \\{\gamma_{d,{skip}} = \frac{\sum_{i,j}S_{dij}}{\sum_{i,j}{{P\left( {{{skip}i},j} \right)}\left( {C_{dij} + S_{dij} + A_{dij}} \right)}}} & (5) \\{{\gamma_{d,{abandon}} = \frac{\sum_{i,j}A_{dij}}{\sum_{i,j}{{P\left( {{{abandon}i},j} \right)}\left( {C_{dij} + S_{dij} + A_{dij}} \right)}}},} & (6)\end{matrix}$

Where numerators in γ_(d,click) (γ⊥^(d,skip))γ⊥(^(d,abandon))) is theactual total number of clicks (skips, abandons) received by item d,whereas the denominator is the expected number of clicks (skips,abandons) an average item is expected to receive when being shown atdifferent positions using different presentation styles the same numberof times as the target item d.

A high value in γ_(d,click) is therefore a positive indicator of itemd's popularity, since it indicates d is driving more clicks than anaverage item. In contrast, γ_(d,skip) and γ_(d,abandon) are supposed tonegatively correlate with item d's popularity. The system and methoddisclosed in this embodiment allow measuring CTR in a relative scale(odds ratio) rather than absolute scale (probability between 0.0 and1.0). For example, suppose it is found that the average CTR at position1 is 1% (probability model) and there is a new article today that hasbeen shown at position 1 for 1,000 times and received 100 clicks (actualoccurrences of clicks). Based on the average CTR, the expectedoccurrences of clicks which this new article shall receive is1000*1%=10. In this case, the click odds is 100/10=10, which indicatesthat this new article obtained 9 times more clicks than expected and istherefore extremely popular.

In the personalized content recommendation system, the counts C_(dij),S_(dij), and A_(dij) may be regularly updated using the latest eventlogs. The values of the click, skip, and abandon odds may also berecomputed for each piece of content, and then fed into the index of theranking system, which uses a machine learned ranking function to scoreitems by combining these three features along with other features usingthe machine learned ranking function.

FIG. 17 is a function block diagram of another example of the userengagement assessment system shown in FIG. 12, according to anembodiment of the present teaching. In this embodiment, a userengagement assessment system 1700 is capable of measuring userengagement from stream depth. The stream depth referred herein may bethe largest number of ranked content items that a user is known to haveviewed in a content stream. The user engagement assessment system 1700in this embodiment enables the engagement measurement of users who maynot provide explicit actions in the personalized content stream 1702.The user engagement assessment system 1700 in this embodiment alsoenables measuring the engagement value of content in the personalizedcontent stream 1702 that may be consumed in part without explicitactions taken by users. Based on the stream depth and position of eachcontent item in the personalized content stream 1702, the userengagement assessment system 1700 scores individual content bygenerating user engagement scores. In this embodiment, the userengagement assessment system 1700 includes a user activity detectionmodule 1704 for detecting user activities that are used for determiningstream depth, including scrolling and abandoning actions, and a userengagement evaluation module 1706 for computing the user engagementscores based on the determined stream depth.

In this embodiment, the user activity detection module 1704 includes ascrolling action detect ion unit 1708 and an abandoning action detectionunit 1710, which are configured to detect scrolling and abandoningactions, respectively. Optionally, a clicking action detection unit 1712may be included to detect explicit actions such as clicking. Theclicking, scrolling and abandoning actions with respect to thepersonalized content stream 1702 have been described before with respectto FIGS. 23-25. Optionally, the detections of the user activities may beperformed over a detection period 1714 for general users or a usercohort 1716, i.e., a group of users determined based on, for example,demographics, visit patterns, on page behaviors, or any other userprofile. It is understood that the detections may be done by any knowntechniques, such as receiving signals from a web beacon or tool bar fromthe user's web browser or any application that renders the personalizedcontent stream 1702.

In this embodiment, the user engagement evaluation module 1706 includesa stream depth calculation unit 1718, a dynamic tipping pointdetermination unit 1720, and a user engagement score calculation unit1722. The stream depth calculation unit 1718 is configured to calculatethe stream depth of the personalized content stream 1702. In oneexample, the stream depth is a function of the following:

1. the number of content items visible by default on the user's pagebefore any actions are taken;

2. the position of the lowest content item explicitly interacted with bythe user, e.g., being clicked, if any; or

3. the position of the lowest content item that the user scrolls to,e.g., detected by a web beacon.

An example of this function may be taking the maximum of the threenumbers above. In one example, assuming no explicit action has beendetected (condition 2) and no scrolling action has been detected(condition 3), then the stream depth is the number of content itemsdisplayed on the screen (condition 1), which is typically determined bythe display screen size and/or display resolution. In another example,if the user scrolls down to content number 18 (condition 3) in thestream without clicking any content (condition 2) and only 4 contentitems can be displayed at the same time on the display screen (condition1), then the stream depth in this case is 18. It is understood that, thestream depth itself may be used directly as a measure of engagement,without regard for intermediate actions taken, number of actions taken,or other behaviors on the page.

In this embodiment, the user engagement score calculation unit 1722 mayprovide a score for individual content in the stream based on the streamdept h from the stream depth calculation unit 1718. In this example, ascoring function may be applied by the user engagement score calculationunit 1722, which is constructed based on the stream depth, tippingpoint, and decay and growth rates 1726. The tipping point may indicatehow far above the abandonment (stream depth) to switch from a positivescore to a negative score, i.e., the zero point in the scoring function.In one example, the tipping point may be a preset tipping point 1724determined based on the number of content items that can be displayed onthe screen at the same time or based on number of content items that canbe presented in a specific area on page for displaying the personalizedcontent stream 1702. For example, the preset tipping point 1724 may bepredetermined based on the statistics of the average users' screen sizeand/or display resolution or the size of the area for displaying thepersonalized content stream 1702. If it is determined that, most of theuser's display screen can have 5 content items displayed simultaneously,then the preset tipping point may be 5. In another example, the tippingpoint may be dynamically determined by the dynamic tipping pointdetermination unit 1720 based on user's behaviors. For example,measurements associated with user's scrolling actions, such as scrollingspeed and acceleration, may be considered by the dynamic tipping pointdetermination unit 1720 to determine a dynamic tipping point to predictthe user's intent behind the action. For instance, if it is detectedthat the user suddenly accelerates the scrolling speed when she/he isreading content number 5 and eventually abandons the stream at contentnumber 8, then the tipping point in this case may be set at contentnumber 5. The details of the scoring function will be described later inFIG. 19.

FIG. 18 is a flowchart of an exemplary process of the user engagementassessment system shown in FIG. 17, according to an embodiment of thepresent teaching. Starting from 1802, a target user or user cohort isfirst determined. The user engagement scores may be designated to aspecific user or a specific user cohort. It is understood that 1802 maynot be necessary if the user engagement scores are for general users. At1804, a target content stream is determined. As the user engagementscores are with respect to each piece of content in the content or thecontent stream itself, the target content and/or content stream may bedetermined first. Moving to 1806, user's non-clicking activities withrespect to the content stream, such as scrolling and abandoning actionsmay be detected. Optionally, the user's clicking actions may also bedetected and recorded.

At 1808, a stream depth is determined based on the detected useractivities. The stream depth referred herein may be the largest numberof ranked content items that a user is known to have viewed in a contentstream. At 1810, a tipping point may be determined based on a presetvalue, for example, the maximum number of content items displayed on thescreen or on the area for displaying the personalized content stream1702. The tipping point may also be dynamically determined based on anestimation of user's intent behind the user's activities, e.g.,accelerating the scrolling speed. At 1812, decay and growth rates arealso determined for constructing a scoring function. The decay rate mayindicate how steeply to decay the scores at the top of the stream asapproaching the tipping point. The growth rate may indicate how steeplyto penalize the content items located between the tipping point and theabandonment point (stream depth).

At 1814, a scoring function is built based on the stream depth, tippingpoint, and decay and growth rates. The scoring function may be a linearfunction, a non-linear function, e.g., an exponential function, alogarithmic function, etc., or any combination thereof. In one example,the goal of the scoring function is to provide a high score to items atthe top of the page, and a negative score to the last several contentitems viewed by the user in the stream. At 1816, user engagement scoresfor each content item in the stream are calculated based on the scoringfunction and their respective positions in the content stream. Forexample, a scoring function defines the relationship between positionsin the stream and the engagement scores (utility values). Once thescoring function is constructed, each content item may be mapped to thescore dimension based on their respective positions in the stream.

Referring now to FIG. 19, an exemplary scoring function 1900 isillustrated. In this example, the content item 1902 at the very top ofthe page is item x. The intuition for the scoring function 1900 is thatif item x is engaging for the user, the user will continue to follow thestream to see item x+1, item x+2, etc. If, however, the user encountersseveral items in a row, e.g., item x+6 1904 (tipping point), item x+7,item x+8, that are of low quality, the user may abandon her/hisinteraction with the stream at the abandonment point (stream depth),e.g., item x+10 1906 (last content item viewed). Thus, the content itemsnear the bottom of the stream, which are near the abandonment point, aregiven large negative scores, while the content items near the top of thestream, which are far away from the abandonment point, are given largehigh positive scores. Also, the content item 1904 at the tipping pointgets zero score by the definition of the tipping point. In the exampleof FIG. 19, the tipping point may be determined as four content itemsabove the abandonment point based on the maximum number of the contentitems that can be displayed at the same time on the screen or the areafor displaying personalized content stream. As the abandonment point inthis example is at the item x+10 1906, the content at the tipping pointis four items above, i.e., item x+6 1904. It is understood that, if itis dynamically monitored that the user starts to accelerate her/hisscrolling speed at item x+6 1904, then item x+6 1904 may also bedynamically set as the tipping point regardless of the screen sizeand/or display resolution. Once the abandonment point (stream depth),tipping point, and decay and growth rates are set, the scoring function1900 is built as shown in FIG. 19, which defines the relationshipbetween the positions in the stream and the engagement scores (utilityvalues). Once the scoring function 1900 is constructed, each contentitem may be mapped to the score dimension based on their respectivepositions in the stream.

FIG. 20 is a function block diagram of still another example of the userengagement assessment system shown in FIG. 12, according to anembodiment of the present teaching. The user engagement assessmentsystem 2000 in this example includes a user device 2010, a user activitydetection module 2040, and a user engagement evaluation module 2050. Onthe user device 2010, a personalized content stream 2012 may bepresented to a user. The personalized content stream 2012 may have beenpersonalized based on some estimated interests of the user. Uponobtaining the personalized content stream 2012, a user may act withrespect to the personalized content stream 2012. The user activitiesfrom one or more users may be detected by the user activity detectionmodule 2040. Based on the user activities, the user engagementevaluation module 2050 may evaluate a user's engagement by calculating auser engagement score.

In this example, the user device 2010 may include a recording unit 2020for recording information associated with activities of the one or moreusers, information associated with the personalized content stream 2012,and/or information associated with the user device 2010. The recordingunit 2020 may include some sub-units, e.g., a screen top recording unit2022, a screen bottom recording unit 2024, a timestamp recording unit2026, and a configuration recording unit 2028. The screen top recordingunit 2022 may record the position at the top of a screen on the userdevice 2010, when an event happens at the user device 2010. The eventmay happen due to a user activity performed with respect to thepersonalized content stream 2012. For example, the screen on the userdevice 2010 may display different pieces of content in the personalizedcontent stream 2012, as the user scrolls through the content in thepersonalized content stream 2012.

The screen bottom recording unit 2024 may record the position at thebottom of the screen on the user device 2010, when the event happens atthe user device 2010. In some embodiments, when some portion of thescreen on the user device 2010 is not visible to a user, the positionrecorded by the screen top recording unit 2022 and screen bottomrecording unit 2024 may be the positions at the top and bottom of avisible portion of the screen, respectively. The timestamp recordingunit 2026 may record a timestamp when the event happens at the userdevice 2010. The configuration recording unit 2028 may recordconfiguration information associated with the user device 2010, e.g.,the size of the screen on the user device 2010, device identification(ID) of the user device 2010, or associated with a user, e.g., a user IDof a logged-in user, browser-cookies for identifying each logged-in ornon-logged-in user.

The user activity detection module 2040 may detect, via the recordingunit 2020 on the user device 2010, events associated a target user oruser cohort 2034, within a detection period 2032. The detection period2032 may be determined based on previous measurements of userengagement. The target user cohort 2034 may include a group of usersdetermined based on, for example, demographics, visit patterns, on pagebehaviors, or any other user profile. In different examples, the eventsdetected by the user activity detection module 940 may be associatedwith some implicit user activities like scrolling and dwelling on thescreen, in comparison to explicit user activities like clicking andsharing.

The user activity detection module 2040 in this embodiment, includes astream ready detection unit 2042, a scroll start detection unit 2044, ascroll end detection unit 2046, and a stream unload detection unit 2048.The stream ready detection unit 2042 may detect a stream ready event,when the content stream 2012 is ready and starts being presented to theuser. The scroll start detection unit 2044 may detect a scroll startevent, when the user starts to scroll through the content in the contentstream 2012. The scroll end detection unit 2046 may detect a scroll endevent, when the user stops scrolling. The stream unload detection unit2048 may detect a stream unload event, when the user discards thecontent stream 2012. The personalized content stream 2012 may bediscarded (abandoned) either due to a closing of the personalizedcontent stream 2012 or a leaving from the personalized content stream2012 to other content. It can be understood that in some examples, oneor more of the events above cannot be detected. For example, the usermay be so deeply engaged in the personalized content stream 2012 thatthe user never unloads the personalized content stream 2012.

The user engagement evaluation module 2050 in this example includes amodel building/updating unit 2052, a user behavior pattern analysis unit2054, a scroll dwell time calculation unit 2058, and a user engagementscore calculation unit 2056. The scroll dwell time calculation unit 2058may calculate a scroll dwell time based on information obtained from theuser activity detection module 2040 and an information interpretationmodel 2053.

Referring now to FIG. 22, an exemplary content stream presented mdifferent time periods is illustrated. In this example, there are eightpieces (d1, d2, . . . , d8) of content in the personalized contentstream 2012 to be presented to the user. The detection period 2032 inthis example, starts at T0 2201, when the personalized content stream2012 is ready, and ends at T7 2208, when the personalized content stream2012 is discarded. There are also three scrolling actions detectedwithin the detection period 2032 in this example, one from T1 2202 to T22203, one from T3 2204 to T4 2205, and another one from TS 2206 to T62207. Within each time period between two scrolling actions, a visibleportion 2280 on the screen of the user device 2010 may cover differentpieces of content. For example, during time period t1, the visibleportion 2280 covers d1, d2, and d3; during time period t2, the visibleportion 2280 covers d2, d3, and d4.

In one example, using an information interpretation model, a scrolldwell time associated with a piece of content may be calculated as thetotal time when the piece of content keeps visible during the detectionperiod 2032, excluding the time of scrolling. For example, as shown inFIG. 22, the scroll dwell time for d1 is t1, i.e., the time period fromT0 from T1, because the visible portion 2280 does not cover d1 any moreafter T1. Similarly, the scroll dwell time for other pieces of contentin the personalized content stream 2012 of this example may becalculated. In this example, the scroll dwell time for d2 is t1+t2; thescroll dwell time for d3 is t1+t2; the scroll dwell time for d4 is t2;the scroll dwell time for d5 is t3; the scroll dwell time for d6 ist3+t4; the scroll dwell time for d7 is t3+t4; the scroll dwell time ford8 is t4.

In case that the personalized content stream 2012 is displayed on a webpage, a web beacon (web bug) may be implemented as the recording unit2020 to track user activities on the stream of web pages. The web beaconmay be an object, e.g., JavaScript, embedded on the web page. A webbeacon may record a timestamp when an event happens and the position ofthe visible portion 2280 when an event happens. For example, referringto the above example in FIG. 22, a web beacon may include“scroll-start:(T3,2)” to record a scroll starts at time T3, when the topof the visible portion 2280 is at d2. In one situation, the web beaconmay record the size of the screen and a device ID of the user device2010. Thus, the size of the visible portion 2280 and the bottom of thevisible portion 2280 at T3 may also be calculated, based on the recordedtop position of the visible portion 2280. In another situation, the webbeacon may directly record a position of the bottom of the visibleportion 2280.

The scroll dwell time calculated based on scrolling actions of a usermay imply a level of user engagement, without an explicit user activitylike clicking or sharing. For example, suppose the eight pieces ofcontent in FIG. 22 are eight summarized articles, each including aheadline and an abstract. Even without any clicks on any summarizedarticles here, if a user dwells a long time at a certain summarizedarticle, the user may have read both the headline and the abstract ofthat summarized article. If a user dwells a short time at a certainsummarized article, the user may have read only the headline of thatsummarized article. If a user scrolls over a certain summarized article,the user may have read neither the headline nor the abstract of thatsummarized article. Thus, based on implicit user activities likescrolling, a level of user engagement may be estimated. It can beunderstood that in some examples, the implicit user activities may becombined with some explicit user activities to estimate a level of userengagement. For example, if a user dwells short after scrolling down toa summarized article, but clicks the summarized article to read thewhole article, then the user may be interested in the article afterreading the headline of the article.

The example in FIG. 22 illustrates a scenario when a user always scrollsdown the personalized content stream 2012. In case that a user scrollsup, the scroll dwell time for a piece of content can be similarlycalculated. Referring to the example in FIG. 22, if a user scrolls up tod1 at T4 2205, and scrolls down to d6, the end of the personalizedcontent stream 2012 at T6 2207, it can be calculated that the scrolldwell time for d1 is t1+t3; the scroll dwell time for d2 is t1+t2+t3;the scroll dwell time for d3 is t1+t2+t3; the scroll dwell time for d4is t2; the scroll dwell time for d5 is 0; the scroll dwell time for d6is t4; the scroll dwell time for d7 is t4; and the scroll dwell time ford8 is t4.

Besides the model used in the example of FIG. 22, the informationinterpretation model 2053 may include other models. For example, whenthe visible portion 2280 covers d2, d3, and d4 for a time period t2,instead of counting t2 into the scroll dwell time for each of the threepieces of content, one third of t2 is counted into the scroll dwell timefor each of d2, d3, and d4. In another example, the time of a scrollingaction may also be counted into the scroll dwell time, especially whenthe scroll speed is low. The scroll speed may be calculated based on thetimestamps and positions of the start and the stop of one scroll.

The model building/updating unit 2052 may build the informationinterpretation model 2053, based on past user activities collected in auser activity log database 2051 in the user engagement evaluation module2050. The user activity log database 2051 may continuously collect useractivities recorded at the user device 2010. The model building/updatingunit 2052 may continuously update the information interpretation model2053 based on updated information at the user activity log database2051, to train the information interpretation model 2053 using machinelearning algorithms. The calculated scroll dwell times may be used aslearning targets for the machine learning algorithms to update theinformation interpretation model 2053.

The user behavior pattern analysis unit 2054 may analyze some userbehavior patterns based on information obtained from the user activitydetection module 2040 and the information interpretation model 2053. Forexample, based on one user's activities, a pattern may be determinedthat the user tends to dwell for long time on a certain type of contentbut tends to dwell shorter or skip on another type of content. Inanother example, based on multiple users' activities, a pattern may bedetermined that a certain position in the personalized content stream2012 may have a shorter scroll dwell time than other positions, acertain position in the visible portion 2280 may have a longer scrolldwell time than other positions, or a certain type of content may tendto have a longer scroll dwell time than other types. The analyzed userbehavior patterns may be used as learning targets for the machinelearning algorithms to update the information interpretation model 2053,e.g., for recommending content that is appropriate with respect to acertain user's behavior pattern.

Based on the analyzed user behavior patterns and the calculated scrolldwell time, the user engagement score calculation unit 2056 maycalculate a user engagement score with respect to each piece of contentin the personalized content stream 2012. The user engagement score mayrepresent a level of user engagement of with respect to thecorresponding piece of content. For example, a long scroll dwell timemay imply a deep user engagement and thus can be transferred to a highuser engagement score. In addition, user behavior patterns may also beconsidered for calculating a user engagement score. For example, supposeusers tend to dwell for long time at the top of the personalized contentstream 2012. Then if two pieces of content have the same scroll dwelltime, one piece of content located at the top of the personalizedcontent stream 2012 may have a lower user engagement score than theother piece of content located at the bottom of the personalized contentstream 2012. In another example, scrolling back and dwelling again atthe same piece of content may indicate an increased engagement of auser, after the user compares the piece of content with others down thestream. The user engagement score may be calculated based on the userbehavior patterns and the calculated scroll dwell time, in combinationwith other parameters discussed earlier, e.g., the stream depth. Forexample, scrolling deep down a content stream and dwelling for a longtime may indicate a high user engagement for the overall content stream.

FIG. 21 is a flowchart of an exemplary process for measuring userengagement, according to an embodiment of the present teaching. At 2102,a target user cohort may be determined. It can be understood that 2102may not be necessary if the user engagement is measured for generalusers. At 2104, a detection period may be determined. At 2106, a targetpiece of content or content stream may be determined. As the userengagement scores are with respect to each piece of content in thecontent or the content stream itself, the target content and/or contentstream may be determined first. The operations of 2102, 2104, and 2106may be performed in serial as shown in FIG. 21, or in parallel. Then at2110, some events may be detected within the determined detectionperiod. The process 1010 may include, for example, detecting a streamready event at 2111, detecting a scroll start event at 2112, detecting ascroll end event at 2113, optionally detecting more scroll events at2114, detecting a stream unload event at 2115, detecting another streamready event 2111, and so on. After each operation in 2110, whether thedetection time has ended may be determined at 2120. If the detectiontime has not ended, the process 2110 may continue. Otherwise if thedetection time has not ended, an information interpretation model may beretrieved at 2130.

The information interpretation model may have been built or updated at2125, based on collected user activities or machine learning algorithmsusing some learning targets. The operation at 2125 may be performedcontinuously. At 2132, a scroll dwell time with respect to the targetcontent or target content stream may be calculated based on theinformation interpretation model and the detected events at 2110. At2134, a pattern of user behaviors with respect to the target content ortarget content stream may be analyzed based on the informationinterpretation model and the detected events at 2110. The calculatedscroll dwell time at 2132 and/or the analyzed user behavior pattern at2134 may be utilized as a learning target for training and updating theinformation interpretation model at 2125. The operations of 2132 and2134 may be performed in serial as shown in FIG. 21, or in parallel. At2136, a user engagement score may be calculated with respect to thetarget content or target content stream based on the scroll dwell timeand the user behavior pattern.

FIGS. 26-28 depict exemplary embodiments of a networked environment inwhich target metric identification is applied, according to differentembodiments of the present teaching. In FIG. 26, an exemplary networkedenvironment 2600 includes the user engagement assessment system 1200,the personalized content recommendation system 1202, the users 1204, acontent portal 2602, a network 2604, and content sources 2606. Thenetwork 2604 may be a single network or a combination of differentnetworks. For example, the network 2604 may be a local area network(LAN), a wide area network (WAN), a public network, a private network, aproprietary network, a Public Telephone Switched Network (PSTN), theInternet, a wireless network, a virtual network, or any combinationthereof. The network 2604 may also include various network accesspoints, e.g., wired or wireless access points such as base stations orInternet exchange points 2604-1, . . . , 2604-2, through which a datasource may connect to the network 2604 in order to transmit informationvia the network 2604.

Users 1204 may be of different types such as users connected to thenetwork 2604 via different user devices, for example, a desktop computer1204-4, a laptop computer 1204-3, a mobile device 1204-1, or a built-indevice in a motor vehicle 1204-2. A user 1204 may send a request andprovide basic user information to the content portal 2602 (e.g., asearch engine, a social media website, etc.) via the network 2604 andreceive personalized content streams from the content portal 2602through the network 2604. The personalized content recommendation system1202 in this example may work as backend support to recommendpersonalized content for the user 1204 to the content portal 2602. Inthis example, the user engagement assessment system 1200 may also serveas backend support for the personalized content recommendation system1202. As described before, the user engagement assessment system 1200may calculate user engagement scores of one or more metrics, such as butnot limited to, click odds, skip odds, abandon odds, stream depths, andstroll dwell times, with respect to each piece of content in a contentstream and provide the user engagement scores to the personalizedcontent recommendation system 1202

The content sources 2606 include multiple third-party content sources2606-1, 2606-2, 2606-3. A content source may correspond to a websitehosted by an entity, whether an individual, a business, or anorganization such as USPTO.gov, a content provider such as cnn.com andfacebook.com, or a content feed source such as Twitter or blogs. Thepersonalized content recommendation system 1202 may access any of thecontent sources 2606-1, 2606-2, 2606-3 to obtain information related tothe users 1204 to construct user profiles and/or collect content tobuild its content pool. For example, the personalized contentrecommendation system 1202 may fetch content, e.g., websites, throughits crawler.

FIG. 27 presents a similarly networked environment as what is shown inFIG. 26 except that the personalized content recommendation system 1202is configured as an independent service provider that interacts with theusers 1204 directly to provide personalized content recommendationservice. In the exemplary networked environment 2700, the personalizedcontent recommendation system 1202 may receive a request with some basicinformation from a user 1204 and provide personalized content streams tothe user 104 directly without going through a third-party content portal2602.

FIG. 28 presents a similarly networked environment as what is shown inFIG. 27 except that the user engagement assessment system 1200 in theexemplary networked environment 2800 is also configured as anindependent service provider to provide user engagement measurementservice for personalized content recommendation.

FIG. 29 depicts a general mobile device architecture on which thepresent teaching can be implemented. In this example, the user device onwhich personalized content is presented is a mobile device 2900,including but is not limited to, a smart phone, a tablet, a musicplayer, a handled gaming console, a global positioning system (OPS)receiver. The mobile device 2900 in this example includes one or morecentral processing units (CPUs) 2902, one or more graphic processingunits (GPUs) 2904, a display 2906, a memory 2908, a communicationplatform 2910, such as a wireless communication module, storage 2912,and one or more input/output (I/O) devices 2914. Any other suitablecomponent, such as but not limited to a system bus or a controller (notshown), may also be included in the mobile device 2900. As shown in FIG.29, a mobile operating system 2916, e.g., iOS, Android, Windows Phone,etc., and one or more applications 2918 may be loaded into the memory2908 from the storage 2912 in order to be executed by the CPU 2902. Theapplications 2918 may include a browser or any other suitable mobileapps for receiving and rendering personalized content streams on themobile device 2900. Execution of the applications 2918 may cause themobile device 2900 to perform the processing as described above, e.g.,in FIGS. 26-28. For example, the display of personalized content to theuser may be made by the GPU 2904 in conjunction with the display 2906.User interactions with the personalized content streams may be achievedvia the I/O devices 2914 and provided to user engagement assessmentsystem 1200 via the communication platform 2910.

To implement the present teaching, computer hardware platforms may beused as the hardware platform(s) for one or more of the elementsdescribed herein. The hardware elements, operating systems, andprogramming languages of such computers are conventional in nature, andit is presumed that those skilled in the art are adequately familiartherewith to adapt those technologies to implement the processingessentially as described herein. A computer with user interface elementsmay be used to implement a personal computer (PC) or other type of workstation or terminal device, although a computer may also act as a serverif appropriately programmed. It is believed that those skilled in theart are familiar with the structure, programming, and general operationof such computer equipment and as a result the drawings should beself-explanatory.

FIG. 30 depicts a general computer architecture on which the presentteaching can be implemented and has a functional block diagramillustration of a computer hardware platform that includes userinterface elements. The computer may be a general-purpose computer or aspecial purpose computer. This computer 3000 can be used to implementany components of the user engagement measurement architecture asdescribed herein. Different components of the system in the presentteaching can all be implemented on one or more computers such ascomputer 3000, via its hardware, software program, firmware, or acombination thereof. Although only one such computer is shown, forconvenience, the computer functions relating to the target metricidentification may be implemented in a distributed fashion on a numberof similar platforms, to distribute the processing load.

The computer 3000, for example, includes COM ports 3002 connected to andfrom a network connected thereto to facilitate data communications. Thecomputer 3000 also includes a central processing unit (CPU) 3004, in theform of one or more processors, for executing program instructions. Theexemplary computer platform includes an internal communication bus 3006,program storage and data storage of different forms, e.g., disk 3008,read only memory (ROM) 3010, or random access memory (RAM) 3012, forvarious data files to be processed and/or communicated by the computer,as well as possibly program instructions to be executed by the CPU. Thecomputer 3000 also includes an I/O component 3014, supportinginput/output flows between the computer and other components thereinsuch as user interface elements 3016. The computer 3000 may also receiveprogramming and data via network communications.

Hence, aspects of the method of measuring user engagement, as outlinedabove, may be embodied in programming. Program aspects of the technologymay be thought of as “products” or “articles of manufacture” typicallyin the form of executable code and/or associated data that is carried onor embodied in a type of machine readable medium. Tangiblenon-transitory “storage” type media include any or all of the memory orother storage for the computers, processors or the like, or associatedmodules thereof, such as various semiconductor memories, tape drives,disk drives and the like, which may provide storage at any time for thesoftware programming.

All or portions of the software may at times be communicated through anetwork such as the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another. Thus, another typeof media that may bear the software elements includes optical,electrical, and electromagnetic waves, such as used across physicalinterfaces between local devices, through wired and optical landlinenetworks and over various air-links. The physical elements that carrysuch waves, such as wired or wireless links, optical links or the like,also may be considered as media bearing the software. As used herein,unless restricted to tangible “storage” media, terms such as computer ormachine “readable medium” refer to any medium that participates inproviding instructions to a processor for execution.

Hence, a machine readable medium may take many forms, including but notlimited to, a tangible storage medium, a carrier wave medium or physicaltransmission medium. Non-volatile storage media include, for example,optical or magnetic disks, such as any of the storage devices in anycomputer(s) or the like, which may be used to implement the system orany of its components as shown in the drawings. Volatile storage mediainclude dynamic memory, such as a main memory of such a computerplatform. Tangible transmission media include coaxial cables; copperwire and fiber optics, including the wires that form a bus within acomputer system. Carrier-wave transmission media can take the form ofelectric or electromagnetic signals, or acoustic or light waves such asthose generated during radio frequency (RF) and infrared (JR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer can read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to aprocessor for execution.

Those skilled in the art will recognize that the present teachings areamenable to a variety of modifications and/or enhancements. For example,although the implementation of various components described above may beembodied in a hardware device, it can also be implemented as a softwareonly solution. In addition, the components of the system as disclosedherein can be implemented as a firmware, firmware/software combination,firmware/hardware combination, or a hardware/firmware/softwarecombination.

While the foregoing has described what are considered to be the bestmode and/or other examples, it is understood that various modificationsmay be made therein and that the subject matter disclosed herein may beimplemented in various forms and examples, and that the teachings may beapplied in numerous applications, only some of which have been describedherein. It is intended by the following claims to claim any and allapplications, modifications and variations that fall within the truescope of the present teachings.

We claim:
 1. A method implemented on at least one machine, each of whichhas at least one processor, storage, and a communication platformconnected to a network for identifying content to recommend to a user,the method comprising: detecting activities of a user with respect to aplurality of content items in a content stream that is displayed on aweb page, wherein the detecting is performed by a web applicationembedded in the web page; obtaining a stream depth measure of thecontent stream based on the detected user activities; generating adepth-dependent function based on the stream depth measure and a tippingpoint dynamically determined based on the detected user activitiesincluding a speed and/or an acceleration of scrolling the contentstream; calculating a user engagement score for each of the plurality ofcontent items in the content stream based on the depth-dependentfunction and a respective depth of the content item in the contentstream; updating a content item pool based on the user engagement scoresof the plurality of content items; and providing, a personalized streamof content items on the web page based on the updated content item pool.2. The method of claim 1, wherein the stream depth measure is a maximumof: a number of content items that are presented in a display screen tothe user; a first depth of a content item in the content stream,relative to a beginning of the content stream, at which the userexplicitly interacts with the content item; and a second depth of acontent item in the content stream, relative to the beginning of thecontent stream, at which the user abandons the content stream.
 3. Themethod of claim 1, wherein the tipping point is dynamically determinedbased on an intent inferred from the detected user activities.
 4. Themethod of claim 1, wherein the user activities include at least one ofscrolling through the content stream and abandoning the content stream.5. The method of claim 1, wherein the tipping point is furtherdetermined based on a number of content items that are presented in adisplay screen to the user.
 6. The method of claim 1, wherein thedepth-dependent function is generated based on at least one of a decayrate and a growth rate.
 7. A non-transitory machine-readable mediumhaving information recorded thereon for identifying content to recommendto a user, wherein the information, when read by the machine, causes themachine to perform the following: detecting activities of a user withrespect to a plurality of content items in a content stream that isdisplayed on a web page, wherein the detecting is performed by a webapplication embedded in the web page; obtaining a stream depth measureof the content stream based on the detected user activities; generatinga depth-dependent function based on the stream depth measure and atipping point dynamically determined based on the detected useractivities including a speed and/or an acceleration of scrolling thecontent stream; calculating a user engagement score for each of theplurality of content items in the content stream based on thedepth-dependent function and a respective depth of the content item inthe content stream; updating a content item pool based on the userengagement scores of the plurality of content items; and providing, apersonalized stream of content items on the web page based on theupdated content item pool.
 8. The medium of claim 7, wherein the streamdepth measure is a maximum of: a number of content items that arepresented in a display screen to the user; a first depth of a contentitem in the content stream, relative to a beginning of the contentstream, at which the user explicitly interacts with the content item;and a second depth of a content item in the content stream, relative tothe beginning of the content stream, at which the user abandons thecontent stream.
 9. The medium of claim 7, wherein the tipping point isdynamically determined based on an intent inferred from the detecteduser activities.
 10. The medium of claim 7, wherein the user activitiesinclude at least one of scrolling through the content stream andabandoning the content stream.
 11. The medium of claim 7, wherein thetipping point is further determined based on a number of content itemsthat are presented in a display screen to the user.
 12. The medium ofclaim 7, wherein the depth-dependent function is generated based on atleast one of a decay rate and a growth rate.
 13. A system having atleast one processor for identifying content to recommend to a user, thesystem comprising: a user activity detection module implemented on theat least one processor and configured to detect activities of a userwith respect to a plurality of content items in a content stream that isdisplayed on a web page, wherein the detecting is performed by a webapplication embedded in the web page; a stream depth calculation unitimplemented on the at least one processor and configured to obtain astream depth measure of the content stream based on the detected useractivities; a user engagement score calculation unit implemented on theat least one processor and configured to: generate a depth-dependentfunction based on the stream depth measure and a tipping pointdynamically determined based on the detected user activities including aspeed and/or an acceleration of scrolling the content stream, andcalculate a user engagement score for each of the plurality of contentitems in the content stream based on the depth-dependent function and arespective depth of the content item in the content stream; and acontent pool update unit implemented on the at least one processor andconfigured to: update a content item pool based on the user engagementscores of the plurality pieces of content, and provide, a personalizedstream of content items on the web page based on the updated contentitem pool.
 14. The system of claim 13, wherein the stream depth measureis a maximum of: a number of content items that are presented in adisplay screen to the user; a first depth of a content item in thecontent stream, relative to a beginning of the content stream, at whichthe user explicitly interacts with the content item; and a second depthof a content item in the content stream, relative to the beginning ofthe content stream, at which the user abandons the content stream. 15.The system of claim 13, wherein the tipping point is dynamicallydetermined based on an intent inferred from the detected useractivities.
 16. The system of claim 13, wherein the user activitiesinclude at least one of scrolling through the content stream andabandoning the content stream.
 17. The system of claim 13, wherein thetipping point is further determined based on a number of content itemsthat are presented in a display screen to the user.
 18. The system ofclaim 13, wherein the depth-dependent function is generated based on atleast one of a decay rate and a growth rate.